MakerWorld BOM to McMaster: building a rule-based hardware matcher

MakerWorld BOM → McMaster-Carr Generator is a small full-stack tool: paste a MakerWorld model URL, scrape the project hardware list, and get an editable table with McMaster-Carr product or search links. The stack is FastAPI + React on the surface; the hard part is everything between “M3-16 mm” in a description and a link that actually opens the right screw length.

This post covers how the repo was scoped at creation and which design problems showed up immediately: bolt length enforcement, tiered match confidence, duplicate rows from multiple BOM sources, a deliberate no-LLM matching policy, notebook-driven development, and API / terms-of-use guardrails for MakerWorld and McMaster-Carr. A closing section covers what landed after the June MVP through July 2026 (phases 6–9 in PLAN.md).

Source repo: jjheffernan/makemcmasterBOM

Project dossier (scope, stack, deliverables): MakerWorld BOM to McMaster Matcher.

What the MVP actually does

The pipeline is intentionally narrow:

MakerWorld URL  →  scrape  →  parse BOM  →  match McMaster  →  editable table  →  CSV export

MakerWorld supplies hardware three ways: embedded JSON in __NEXT_DATA__, prose in the project description, and optional CSV/XLSX attachments. The backend normalizes all of that into a shared Part model, runs deterministic McMaster resolution, and returns a Project the React editor can fix by hand. There is no database in the MVP; projects live in an in-memory store until the process restarts.

That scope cap matters. The tool is not a general procurement system, not an ERP connector, and not an AI shopping assistant. It is a BOM hygiene layer for 3D-print projects where the author already listed hardware but did not link SKUs.

Keeping the problem correctly scoped

Feature creep arrived on day one: price quotes, cart checkout, multi-vendor comparison, automatic checkout on McMaster, LLM “smart matching” for vague lines like “some screws.”

Each of those expands legal surface, needs credentials, or hides errors behind probabilistic guesses. The execution plan (PLAN.md) keeps phases explicit: ingest MakerWorld reliably, parse to structured parts, attach McMaster links with honest confidence, ship an editor + CSV export, then iterate with regression tests.

Deferred by design:

• Persistent projects / user accounts
• Automatic McMaster ordering
• Hosted third-party McMaster APIs that require external keys
• LLM-based description parsing

The product promise is narrower and more defensible: turn a MakerWorld BOM into an editable McMaster-oriented spreadsheet the human still verifies.

No LLM matching strategy

MakerWorld descriptions are messy, but they are not unstructured noise. They repeat patterns: 13 x M3-16 mm, assembly notes (Left module: 4 x M3-8), quantity prefixes, and section headers like “Material required.”

The description parser is explicitly rule-based, no AI (parsers/makerworld/description.py). Extraction uses:

• Section keyword detection (BOM, Material required, fastener headings)
• Shared quantity regexes in parsers/helpers/bom_quantities.py
• Hardware signal checks (has_hardware_signal) to skip filament and printed parts
• Line-level parsing without model inference

Matching McMaster rows follows the same philosophy. Resolution is a tiered offline resolver (vendors/mcmaster/tiers.py): curated catalog JSON, length tables for common SKUs, embedded part numbers in text, filtered browse URLs with metric thread + length facets, then broader category or site search. Optional live enrichment (official B2B API or Playwright browse tables) sits behind feature flags and never replaces the offline path.

Why avoid LLMs here:

When a line is ambiguous, the UI shows low confidence and a search link, not a fabricated SKU. Humans remain the final matcher.

Enforcing bolt length (and diameter) rules

Catalog hits feel authoritative until a MakerWorld line says M3×16 mm and the matched SKU is a 20 mm cap screw with high heuristic score. Fasteners need post-match verification, not just string similarity.

hardware_spec.py extracts metric diameter and length from many prose shapes:

• Joined forms: M3x16, M3-16 mm, M3×16 mm
• Split forms: M4 screw 40 mm
• Trailing length: socket head screw 16 mm

primary_fastener_spec() prefers original_name over polluted specification fields (MakerWorld often stuffs assembly notes into the wrong column).

After a catalog match, hardware_match_verify.py compares BOM specs to catalog title specs:

For screws without length, the matcher prefers filtered browse URLs with thread-size~m3/ and length~16-mm/ path facets over a blind catalog SKU. Ranking deliberately elevates filtered browse (~0.90) above a weak catalog guess (~0.72–0.84) when metric thread and length are known.

That is the core bolt-length lesson: treat length as a first-class constraint, not a post-search filter the user discovers after clicking.

Tiered matching confidence

A single “confidence” number hides too much. The repo uses match tier + numeric score + editor status.

Offline tiers (McMaster adapter docs):

Site-wide “Standard Components” search was dropped from the BOM matcher — too broad, and it polluted rankings with non-hardware junk. Non-searchable lines (filament, printed parts) get trimmed before match.

Heuristic scoring adds signals (hardware keywords, digits, spec field present), but catalog hits no longer auto-score 1.0. Verification can restore 1.0 on verified or corrected; mismatches clamp confidence.

resolve_match_status() maps tier + score to UI labels (likely, possible, unlikely). The editor highlights rows that need eyes before export. Later work also surfaces structured guesses (same-size vs wider-scope alternatives) so the UI can group alternatives without faking a single SKU.

Optional live paths are separate tiers:

• Official Product Information API (MCMASTER_API_ENABLED): B2B client cert, subscription limits, enriches descriptions and product status; credentials never reach the browser (API docs).
• Browse table resolve / enrich (MCMASTER_BROWSE_RESOLVE_ENABLED and related flags): Playwright intercept of ProdPageWebPart.aspx JSON for finish variants and listing hydration; slow, opt-in, integration-gated.

Default CI still prefers offline tiers, so tests stay deterministic; local --debug can enable enrich for UI work.

Duplicate parts across BOM sources

MakerWorld often lists the same hardware twice: embedded BOM, description prose, and an uploaded spreadsheet. Without a merge policy, quantities double and McMaster matching runs twice on identical lines.

merge_parts() in parsers/helpers/parts_merge.py combines lists with a simple rule:

key = (part.original_name.strip().lower(), part.quantity)

First occurrence wins; later duplicates drop. That is deliberately conservative:

• Same name, same quantity → duplicate (skip)
• Same name, different quantity → kept as separate rows (may be intentional: sub-assemblies)
• Slightly different strings (M3x16 mm vs M3-16 mm socket head) → not deduplicated yet (normalization is a follow-up)

Source priority matured in July: when the description has an explicit BOM section (Bill of Materials, BOM:), merge_description_with_embedded() prefers those lines ahead of MakerWorld’s embedded supply list. Inferred hardware phrases scattered through prose still lose to embedded when both exist — author’s labeled list beats silent JSON noise.

Open questions that remain:

• Should normalized fastener specs merge across wording variants?
• Should quantities sum when duplicates clearly refer to the same SKU?

The answer still favors no silent quantity math until more regression fixtures prove it safe. Wrong totals are worse than duplicate rows a human deletes in the editor.

Notebook-driven development

The repo treats Jupyter notebooks as the first implementation surface, not demo-only artifacts. Numbered notebooks mirror pipeline stages:

Rules from the notebook-driven pipeline skill:

1. Prototype in the stage notebook with fixtures in data/ or curated URLs in data/regression_urls.json
2. Promote stable functions into backend/services/
3. Re-import services from notebooks (no forked logic)
4. Add pytest coverage before exposing new behavior on FastAPI routes

notebook_utils.py adds Jupyter-only helpers (safe_scrape, safe_import_project, offline fallbacks). The website always runs the live pipeline; notebooks may cache when MakerWorld rate limits bite.

./scripts/dev.sh starts API, Vite, and JupyterLab together so the /notebooks page and import UI exercise the same code path.

API and terms-of-service considerations

Two external sites define the legal and operational envelope.

MakerWorld (scraping)

All MakerWorld HTTP traffic is server-side (httpx, optional Playwright). The browser never fetches model pages directly (CORS and logic exposure). Outbound politeness is enforced in rate_limit.py:

SCRAPER=auto tries httpx first, falls back to headless Chromium on 403/proxy failures. Regression URLs and fixtures reduce how often CI hammers live pages.

Operational stance: cache HTML for tests, throttle in dev, disable rate limits only locally (RATE_LIMIT_ENABLED=0), and return 200 with warnings when a project has no BOM instead of retry-storming ambiguous pages.

McMaster-Carr (linking, optional API, optional browse)

McMaster linking is mostly URL construction + curated catalog data, not bulk scraping in the default mode.

Constraints documented in-repo:

• Official API requires approved B2B account, client certificate, per-part subscription limits, and 24h tokens. Enabled only when credentials exist; responses stay server-side (Product Information API).
• Browse table resolution uses Playwright against public product tables. Docs note McMaster expects automated load stay tied to purchasing decisions: keep outbound intervals, disable browse in CI, prefer offline catalog + filtered browse URLs by default.
• Guardrail tests scan for credential leaks in API responses and tracked files.

Third-party hosted McMaster wrappers were avoided so the vendor adapter template stays self-contained and keyless for MVP users.

Stack snapshot

Full architecture: docs/architecture.md.

What shipped vs what is next

Phases 1–5 (ingest → parse → match → editor → export) locked the June MVP. PLAN.md now marks phases 1–9 complete as of July 2026. Phase 10 is the live backlog.

July 2026: matcher maturity, taxonomy, enrich, and editor UX

After the MVP shipped, the next risk was not “more AI” — it was wrong aisle, wrong finish, and stale category maps. Work concentrated on category routing, offline data upkeep, optional live enrich, and editor affordances that keep uncertainty visible.

McMaster category routing (phase 8)

Offline matching expanded past generic screw paths:

In-house browse scrape (browse_scrape / browse_parse / browse_fetch) replaced the temporary upstream scraper dependency (archived under docs/archive/).

Monthly taxonomy crawl

McMaster family tiles move. The repo now refreshes that map offline, not on every import:

• GitHub Actions workflow on the 1st of each month (plus ./scripts/run_monthly_taxonomy_crawl.sh locally)
• Polite batch crawl of Fastening & Joining child browse pages (ProdPageWebPart.aspx JSON, 5–6 s delay)
• Writes data/mcmaster_site_taxonomy.json; can sync metacategory slugs; opens a PR when diffs appear
• Docs: mcmaster-taxonomy.md

Imports stay light. Humans review taxonomy PRs before promoting high-value families into mcmaster_categories.json.

Description BOM wins when the author labeled it

MakerWorld embedded supply JSON often includes filament and incomplete hardware. When the description contains an explicit BOM header, merge order is now description first, then embedded leftovers (merge_description_with_embedded). Inline Bill of Materials: … prose is split onto its own lines so the section detector fires. Inferred hardware sentences without a labeled section still defer to embedded.

Enrich stage, pricing, and match feedback (phase 9)

The UI caught up to the richer matcher:

• Import progress shows an enrich stage when live browse/API hydration runs
• Pricing tab with pack-aware line totals (POST /api/bom/sync-pricing — still on the security hardening backlog for rate limits)
• Match warnings + grouped guess scopes in the editor
• BOM section headings and drag-to-reorder within sections
• Category-specific hardware check hint tooltips
• Report match error with reporter email → local JSONL plus optional email / GitHub issue / webhook fan-out (feedback-dispatch.md)

Regression harness grew with it: 350+ offline pytest cases, query-accuracy fixtures, guardrail scans, and CI on push/PR.

Docs and ops

Security threat model + deployment checklist, cross-linked architecture/API/testing guides, cleanup/uninstall scripts, and a clearer docs/ index. The no-LLM matching rule did not change.

Phase 10 backlog (selected)

1. Filtered browse roots for more fastener families still on category-search (hex_bolt, threaded_rod, set_screw, …)
2. Expand matcher categories beyond Fastening (power transmission, sealing, fittings)
3. Cross-check hardware_spec extractions against McMaster API Specifications[] when credentials exist
4. Frontend component tests + gated E2E import smoke
5. Security hardening before any public deploy: strict host URL validation, rate-limit sync-pricing, upload size caps, auth on the API
6. Persistent project store if multi-user use appears

The repo exists because printable project pages already contain BOMs, but not purchasable links with honest uncertainty. The June bet still holds: deterministic parsing + tiered McMaster URLs + length verification. July’s contribution is making that bet survive more fastener kinds, stale category trees, and shop-floor editor habits — without turning matching into a black-box model.

If that workflow matches a shop process, clone makemcmasterBOM, run ./scripts/dev.sh, and start in 04_match_mcmaster.ipynb before touching the matcher.