CrawlingData Collection

Price & Stock Monitoring Automation System

A monitoring system that automatically collects prices and stock across multiple sales channels every day, sends alerts on changes, and compares everything at a glance on a dashboard.

Client
Retail/Commerce Client
Duration
5 weeks
Cost
~7M KRW
Status
Completed

Tech Stack

PythonPlaywrightPostgreSQLNext.jsAWS SES

Project Overview

A client selling products across several channels hit a wall checking each channel's price and stock manually. They asked for a system that collects publicly listed price and stock data daily, alerts them the moment something changes, and lets them compare it all on a dashboard.

Challenges

1

Each channel had a different page structure, requiring per-channel collection logic

2

Some pages rendered content with JavaScript, so plain requests returned nothing

3

We had to prevent the crawler from dying silently and leaving days of missing data

Solutions

1

Browser-automation collection

Dynamic-rendering pages were handled with Playwright browser automation, while static pages used a lighter method — securing both server cost and stability. Collection intervals were designed to avoid burdening the target servers.

2

Failure detection and alerts

The system emails immediately if collection fails or data looks abnormal, blocking the classic "dies quietly, discovered later" crawler accident at the architecture level.

3

Comparison dashboard

Collected data is compared by channel and product, with price-trend visualization. Items with the largest swings surface at the top so the client sees exactly where action is needed.

Key Features

Daily auto-collection

Gathers price and stock per channel automatically at a set time.

Change alerts

Emails immediately when meaningful price or stock changes occur.

Comparison dashboard

Compares status and trends by channel and product on one screen.

Data export

Download collected data as Excel/CSV for separate analysis.

Project Results

The daily manual checking disappeared into automation, freeing the time spent on monitoring. Change alerts mean response timing is never missed, and failure alerts keep it running with no data gaps.

Client Review

It was a data crawling request with many complex conditions, but they understood the requirements accurately from the first consultation. The data was delivered in a clean, organized format and was highly usable.

J
JLG
Verified marketplace review

Need a similar project?

Enter your requirements and AI will instantly analyze your estimated quote.

Get a Free Quote

Explore Other Projects