This project involved building a data pipeline that scrapes product and sales data from major Korean e-commerce platforms like Coupang, Naver Store, and Today’s House. The data was transformed and delivered to an interactive dashboard, while optimizing query performance, achieving a significant 300% speed improvement.

Key Features

• Data Scraping: Regular collection of product and sales data from Coupang, Naver Store, and Today’s House.
• ETL Pipeline: An automated pipeline to extract, transform, and load large-scale data seamlessly.
• Query Optimization: Solving performance issues by optimizing SQL queries, boosting data processing speed by 300%.

‍

Challenges

The main challenge was optimizing the query performance of the original system, which led to delays in processing large datasets. Additionally, integrating data from multiple e-commerce platforms and ensuring the accuracy of data collection required careful planning.

‍

Development process

We selected scraping methods best suited for each platform (API, cookies, Selenium), then optimized inefficient SQL queries and implemented indexing to significantly enhance performance. We also worked on automating the ETL pipeline for scalability.

‍

Tech stacks

‍

The outcome

The optimized data pipeline now handles large datasets more efficiently, improving processing speeds by 300%. With regular and reliable data scraping, the team can now leverage up-to-date insights and make data-driven decisions in real time.