Track A — Catalog Operations
Scrape every shop, review every product — the operational runbook
This track is not coding. It's the operational work of building a trustworthy product catalog: configuring shops, running scrapes, and reviewing what comes back. It's also the long pole to launch — reviewing thousands of products takes calendar weeks no matter how fast the code work goes.
How the pipeline works today
Shops are configured in the database via the admin UI (/admin/scraper), not in code:
- A
ScraperShopholds the shop name plus its BrightData PLP and PDP collector ids. - Each
ScraperSourceunder it is one category listing URL (e.g./fpv/moteurs→MOTOR). - A PLP scrape creates
ScrapedProductrows in statuspending. - You triage them in the review queue: approve →
ready, reject →skip. - A PDP scrape enriches
readyproducts (full description, images, specs) →pdp_done(orantibot_blockedon failure — retriable). - Import turns a
pdp_donescraped product into a catalogProduct, applying the per-category spec mappings (ScraperSpecMapping— scraped key →SpecType). - Imported products land in the product review queue (
/admin/products) where name, brand, category and specs get validated before the product is live and searchable.
Stale spec
The scrapers roadmap page still describes a site-configs.ts/SITE_REGISTRY architecture — that predates the BrightData refactor. Trust this page for the operational flow.
1. Foundation pass (do once, before mass review)
Everything you review inherits the category/spec structure that exists at review time. Fixing it later means re-touching reviewed products, so freeze it first (part of M0):
- Category tree final: frame, motor, FC, ESC, AIO, VTX, camera, antenna, receiver, radio, goggles, battery/charger, props, GPS, accessories. Decide now what's out of scope (e.g. spare screws, apparel).
- Spec templates per category (
/admincatalog screens): motor → KV/stator/shaft; VTX → power/band; battery → cells/capacity/C-rating; frame → size/wheelbase… Only specs buyers filter on. - Spec mappings seeded per shop × category: run a small test scrape first (below), look at the raw keys each shop emits, map them once. Unmapped keys are silently lost at import.
- Naming convention written down:
Brand Model Variant(e.g. "iFlight XING2 2306 1855KV"), no shop suffixes ("- Drone-FPV-Racer"), no marketing text. Consistency here is what makes dedup and matching possible.
2. Per-shop runbook
Repeat for each shop, in rollout order (§4). Don't parallelize shops — finish one before starting the next, so mistakes stay contained.
Configure & smoke-test
Verify the shop's collector ids and create one ScraperSource per category. Run a PLP scrape on one category and spot-check ~10 results: name, price, currency, image, availability all sane? If the shop needs new BrightData collectors, build and test those first.
Full PLP scrape
Trigger all sources. Stagger them (politeness, and BrightData cost). Confirm each source's lastRunAt updates and the pending count looks like the shop's real catalog size.
Triage the review queue
Work through pending products with the bulk actions, using the triage rules (§3). Target: 0 pending for the shop before moving on. This is the bulk of the work — batch it daily.
PDP pass
Scrape details for everything ready. Retry antibot_blocked items once or twice on a later day; if a batch stays blocked, note it and move on — don't burn days on stragglers.
Import & product review
Import pdp_done products. In /admin/products, validate each: name per convention, brand set, category right, specs populated (if specs are systematically empty for a category, the spec mapping is wrong — fix it and re-import that category, not product-by-product).
Match & dedup against existing catalog
For shops after the first: scraped products that already exist as catalog products must be linked as an additional Shop price entry, not imported as duplicates. Auto-matching is weak today (see §6) — budget manual matching time for shop #2 onward.
Verify on the live site
Browse the public catalog for this shop's categories: images load, prices/availability shown, search finds them, filters work. Then record the shop as done in §4.
3. Triage rules
Decide these once so every batch is judged the same way:
- Approve: single identifiable product, in a supported category, matchable to a brand.
- Skip: bundles/kits ("frame + motors combo"), shop gift cards, apparel, repair services, products with no usable image, categories you declared out of scope.
- Variants (KV options, prop colors): one catalog product per meaningfully different variant (a 1855KV and 2555KV motor are different products; prop colors are not). Follow the naming convention.
- Duplicates within a shop (same product in two source categories): approve once, skip the rest.
- Use bulk select per brand/keyword to clear obvious groups fast; hand-review only the ambiguous tail.
4. Shop rollout order
| # | Shop | Status | Notes |
|---|---|---|---|
| 1 | drone-fpv-racer.com | ⬜ configured, needs full pass | Largest FR shop — do first, it sets the catalog baseline |
| 2 | studiosport.fr | ⬜ configured, needs full pass | First real test of cross-shop matching |
| 3 | lacameraembarquee.fr | ⬜ needs collectors | High priority — big FR catalog |
| 4 | rcmodelisme.fr | ⬜ needs collectors | Medium |
| 5 | flashrc.com | ⬜ needs collectors | Medium |
| — | TBS / getfpv / dfrshop | ⬜ | Post-launch — see scrapers spec |
Update the Status column as you go — this table is the tracker.
5. Acceptance bar
Per shop: 0 pending scraped products; ≥95% of ready reached pdp_done; imported products all through product review; no shop-suffixed names in the first 100 search results.
Catalog launch bar (gates M2):
- Every core category (frame, motor, FC/ESC/AIO, VTX, camera, goggles, radio, battery) has enough products that a buyer creating a deal finds their gear — spot-check by building 5 realistic used-quad listings end-to-end.
- No duplicate products in the top-100 of any core category.
- Cross-shop: products carried by both shops show both prices.
Throughput planning: expect 1,000–3,000 products per shop. At a sustainable 100–150 reviews/day, a shop is roughly 2–3 weeks of daily batches — that's why Track A starts at M0 and never pauses.
6. Coding dependencies of this track
Small Track B/C tasks that this track needs — schedule them into M2:
| Task | Why | Where specified |
|---|---|---|
| Re-upload scraped images to S3 at import | Catalog currently hotlinks shop images; also lets us drop the hostname: "**" open-proxy image pattern | Launch checklist |
PriceHistory model + record on syncPrice | Cheap to start now; powers price trends & alerts later | Scrapers spec |
| Better auto-matching (fuzzy name + brand/category, confidence in admin UI) | Shop #2+ matching is manual today | Scrapers spec |
| Cron scheduling (daily price re-scrape, weekly catalog) | Manual triggering doesn't survive launch | Operations §3 |
| Scraper Sentry + health visibility | Failed jobs currently only visible in Redis | Operations §6 |
Until cron lands: manually re-run price scrapes weekly on completed shops so availability/prices don't rot while you review the next shop.