Shop Scrapers

Current state

Shops are configured in the database via the admin UI (/admin/scraper): a ScraperShop holds BrightData PLP/PDP collector ids, each ScraperSource is one category listing URL, and scraping runs as BullMQ jobs in apps/scraper against BrightData. Two shops configured today:

drone-fpv-racer.com (DFR)
studiosport.fr

Adding a shop means building/testing BrightData collectors for its PLP and PDP pages, then registering the shop + sources in the admin UI — the operational runbook is in Catalog operations.

Target shops (FR market)

Shop	Platform	Priority
drone-fpv-racer.com	PrestaShop	✅ Done
studiosport.fr	OASIS	✅ Done
lacameraembarquee.fr	Custom/WooCommerce	High — large catalog, popular FR shop
rcmodelisme.fr	PrestaShop	Medium — big catalog, multi-category
flashrc.com	PrestaShop	Medium — good FPV section
team-blacksheep.com	Custom	Medium — official TBS store
getfpv.com / racedayquads.com	Shopify	Low — US-based, USD prices
dfrshop.com	PrestaShop	Low — sub-brand of DFR

Per-shop setup: one BrightData PLP collector + one PDP collector (built and tested on the BrightData side), then in the admin UI a ScraperShop with those collector ids and one ScraperSource per category URL (e.g. /fpv/moteurs → MOTOR).

Price history

Track price changes over time — powers price trends on product pages and the price-history charts roadmap item. Cheap to start recording now, valuable later.

model PriceHistory {
  id        Int      @id @default(autoincrement())
  shopId    Int
  shop      Shop     @relation(fields: [shopId], references: [id])
  price     Decimal
  currency  String
  available Boolean
  scrapedAt DateTime @default(now())

  @@index([shopId, scrapedAt])
}

Product matching

Scraping creates ScrapedProduct records needing matching to Product entries. Improve auto-matching: fuzzy name matching (Levenshtein), brand + category + name combo, SKU/EAN when available, and an admin UI showing suggested matches with confidence scores.

Scheduling & hygiene

Daily scrape for price/availability; weekly full catalog scrape for new products; immediate re-scrape on admin trigger.
Stagger scrapes + delays between requests (polite scraping); respect robots.txt, identify the bot — see the scraping posture note in the launch checklist.
Scraper health dashboard (last run, success rate, product count) + alert on repeated failures (DOM structure changed) — pairs with Sentry on the scraper.

Tasks

La Camera Embarquee, rcmodelisme.fr, flashrc.com configs
PriceHistory model + recording on syncPrice
Improved auto-matching + confidence scores in the admin UI
Cron scheduling (daily price, weekly catalog); rate limiting
Health dashboard + failure alerts