KwadMarket Docs
Road to Production

Track A — Catalog Operations

Scrape every shop, review every product — the operational runbook

This track is not coding. It's the operational work of building a trustworthy product catalog: configuring shops, running scrapes, and reviewing what comes back. It's also the long pole to launch — reviewing thousands of products takes calendar weeks no matter how fast the code work goes.

How the pipeline works today

Shops are configured in the database via the admin UI (/admin/scraper), not in code:

  1. A ScraperShop holds the shop name plus its BrightData PLP and PDP collector ids.
  2. Each ScraperSource under it is one category listing URL (e.g. /fpv/moteursMOTOR).
  3. A PLP scrape creates ScrapedProduct rows in status pending.
  4. You triage them in the review queue: approve → ready, reject → skip.
  5. A PDP scrape enriches ready products (full description, images, specs) → pdp_done (or antibot_blocked on failure — retriable).
  6. Import turns a pdp_done scraped product into a catalog Product, applying the per-category spec mappings (ScraperSpecMapping — scraped key → SpecType).
  7. Imported products land in the product review queue (/admin/products) where name, brand, category and specs get validated before the product is live and searchable.

Stale spec

The scrapers roadmap page still describes a site-configs.ts/SITE_REGISTRY architecture — that predates the BrightData refactor. Trust this page for the operational flow.

1. Foundation pass (do once, before mass review)

Everything you review inherits the category/spec structure that exists at review time. Fixing it later means re-touching reviewed products, so freeze it first (part of M0):

  • Category tree final: frame, motor, FC, ESC, AIO, VTX, camera, antenna, receiver, radio, goggles, battery/charger, props, GPS, accessories. Decide now what's out of scope (e.g. spare screws, apparel).
  • Spec templates per category (/admin catalog screens): motor → KV/stator/shaft; VTX → power/band; battery → cells/capacity/C-rating; frame → size/wheelbase… Only specs buyers filter on.
  • Spec mappings seeded per shop × category: run a small test scrape first (below), look at the raw keys each shop emits, map them once. Unmapped keys are silently lost at import.
  • Naming convention written down: Brand Model Variant (e.g. "iFlight XING2 2306 1855KV"), no shop suffixes ("- Drone-FPV-Racer"), no marketing text. Consistency here is what makes dedup and matching possible.

2. Per-shop runbook

Repeat for each shop, in rollout order (§4). Don't parallelize shops — finish one before starting the next, so mistakes stay contained.

Configure & smoke-test

Verify the shop's collector ids and create one ScraperSource per category. Run a PLP scrape on one category and spot-check ~10 results: name, price, currency, image, availability all sane? If the shop needs new BrightData collectors, build and test those first.

Full PLP scrape

Trigger all sources. Stagger them (politeness, and BrightData cost). Confirm each source's lastRunAt updates and the pending count looks like the shop's real catalog size.

Triage the review queue

Work through pending products with the bulk actions, using the triage rules (§3). Target: 0 pending for the shop before moving on. This is the bulk of the work — batch it daily.

PDP pass

Scrape details for everything ready. Retry antibot_blocked items once or twice on a later day; if a batch stays blocked, note it and move on — don't burn days on stragglers.

Import & product review

Import pdp_done products. In /admin/products, validate each: name per convention, brand set, category right, specs populated (if specs are systematically empty for a category, the spec mapping is wrong — fix it and re-import that category, not product-by-product).

Match & dedup against existing catalog

For shops after the first: scraped products that already exist as catalog products must be linked as an additional Shop price entry, not imported as duplicates. Auto-matching is weak today (see §6) — budget manual matching time for shop #2 onward.

Verify on the live site

Browse the public catalog for this shop's categories: images load, prices/availability shown, search finds them, filters work. Then record the shop as done in §4.

3. Triage rules

Decide these once so every batch is judged the same way:

  • Approve: single identifiable product, in a supported category, matchable to a brand.
  • Skip: bundles/kits ("frame + motors combo"), shop gift cards, apparel, repair services, products with no usable image, categories you declared out of scope.
  • Variants (KV options, prop colors): one catalog product per meaningfully different variant (a 1855KV and 2555KV motor are different products; prop colors are not). Follow the naming convention.
  • Duplicates within a shop (same product in two source categories): approve once, skip the rest.
  • Use bulk select per brand/keyword to clear obvious groups fast; hand-review only the ambiguous tail.

4. Shop rollout order

#ShopStatusNotes
1drone-fpv-racer.com⬜ configured, needs full passLargest FR shop — do first, it sets the catalog baseline
2studiosport.fr⬜ configured, needs full passFirst real test of cross-shop matching
3lacameraembarquee.fr⬜ needs collectorsHigh priority — big FR catalog
4rcmodelisme.fr⬜ needs collectorsMedium
5flashrc.com⬜ needs collectorsMedium
TBS / getfpv / dfrshopPost-launch — see scrapers spec

Update the Status column as you go — this table is the tracker.

5. Acceptance bar

Per shop: 0 pending scraped products; ≥95% of ready reached pdp_done; imported products all through product review; no shop-suffixed names in the first 100 search results.

Catalog launch bar (gates M2):

  • Every core category (frame, motor, FC/ESC/AIO, VTX, camera, goggles, radio, battery) has enough products that a buyer creating a deal finds their gear — spot-check by building 5 realistic used-quad listings end-to-end.
  • No duplicate products in the top-100 of any core category.
  • Cross-shop: products carried by both shops show both prices.

Throughput planning: expect 1,000–3,000 products per shop. At a sustainable 100–150 reviews/day, a shop is roughly 2–3 weeks of daily batches — that's why Track A starts at M0 and never pauses.

6. Coding dependencies of this track

Small Track B/C tasks that this track needs — schedule them into M2:

TaskWhyWhere specified
Re-upload scraped images to S3 at importCatalog currently hotlinks shop images; also lets us drop the hostname: "**" open-proxy image patternLaunch checklist
PriceHistory model + record on syncPriceCheap to start now; powers price trends & alerts laterScrapers spec
Better auto-matching (fuzzy name + brand/category, confidence in admin UI)Shop #2+ matching is manual todayScrapers spec
Cron scheduling (daily price re-scrape, weekly catalog)Manual triggering doesn't survive launchOperations §3
Scraper Sentry + health visibilityFailed jobs currently only visible in RedisOperations §6

Until cron lands: manually re-run price scrapes weekly on completed shops so availability/prices don't rot while you review the next shop.

On this page