python Case study
Digikala Excel Inventory Automation
Checking a few thousand products against their marketplace listings by hand is a job nobody finishes twice. The interesting engineering here is not the scraping, it is making a long run survivable: it caches, it logs progress as it goes, and it can be stopped and resumed.
The business problem
The inventory spreadsheet and the marketplace listings drift apart constantly, and reconciling them by hand takes days and produces different answers depending on who did it. Automating it runs into the real constraint: a run over thousands of products takes hours, and anything that fails halfway through with nothing written is worse than useless.
What I delivered
- A Python pipeline that reads the inventory spreadsheet with pandas and writes a status workbook back out. Input and output are both files the operations team already works in.
- A persistent JSON cache of every lookup, which is what makes a run resumable rather than restartable.
- An append-only progress CSV written as the run proceeds, so partial results survive an interruption and the run can be monitored while it is going.
- A Playwright browser path for listings that require rendering, with requests and BeautifulSoup for the ones that do not.
- A search-and-verify strategy with an exact-slug match and a search fallback, rather than trusting a single lookup route.
- Randomised pacing between requests, because a scraper that hammers a marketplace gets blocked and then the whole run is worthless.
- A version stamp in the source, so a given output file can be traced to the logic that produced it.
Technical approach
- The cache is the core design decision. Every resolved product is written to disk immediately, which turns a fragile multi-hour run into something that can be interrupted freely.
- The pipeline appends progress as it goes instead of holding it in memory until the end. A crash at 90 percent should cost the last product, not the whole run.
- Two fetch strategies rather than one. I try the cheap HTTP path first and reach for the browser only where it is genuinely needed, since launching a browser per product would make the run impossible.
- I verify every match rather than assume it. An exact slug match wins, and a search fallback runs when that fails, since a confidently wrong match is worse than no match.
- Randomised delays are a correctness requirement here, not politeness. A blocked run produces bad data, not slow data.
Result and evidence
Reconciliation became a run rather than a project. The output is a status workbook the operations team opens directly, and the cache means a rerun after a partial failure costs minutes rather than hours.
Commercial value
This is the difference between a check that happens quarterly because it is painful and one that happens whenever it is needed.
Readable implementation brief
implementation_brief {
project: "Digikala Excel Inventory Automation"
stack: "Python, pandas, requests, BeautifulSoup, Playwright"
io: "inventory .xlsx in, status .xlsx out"
resumability: "persistent JSON cache + append-only
progress CSV, written as the run proceeds"
fetch: "HTTP first; headless browser only where required"
matching: "exact slug preferred, search fallback, verified"
pacing: "randomised delays; a blocked run is bad data"
traceability: "patch version stamped in source"
}What this project shows
Resumability is what I would want noticed. Anyone can write the scraping; designing a long-running job so an interruption is cheap is the part that decides whether it is ever used twice.
I apply the same instinct to cost: take the cheap fetch path, and escalate to a browser only when the page forces it.