MakerWorld BOM to McMaster: building a rule-based hardware matcher

MakerWorld BOM → McMaster-Carr Generator is a small full-stack tool: paste a MakerWorld model URL, scrape the project hardware list, and get an editable table with McMaster-Carr product or search links. The stack is FastAPI + React on the surface; the hard part is everything between "M3-16 mm" in a description and a link that actually opens the right screw length.

This post covers how the repo was scoped at creation and which design problems showed up immediately: bolt length enforcement, tiered match confidence, duplicate rows from multiple BOM sources, a deliberate no-LLM matching policy, notebook-driven development, and API / terms-of-use guardrails for MakerWorld and McMaster-Carr.

Source repo: jjheffernan/makemcmasterBOM

Project dossier (scope, stack, deliverables): MakerWorld BOM to McMaster Matcher.

What the MVP actually does

The pipeline is intentionally narrow:

MakerWorld URL  →  scrape  →  parse BOM  →  match McMaster  →  editable table  →  CSV export

MakerWorld supplies hardware three ways: embedded JSON in __NEXT_DATA__, prose in the project description, and optional CSV/XLSX attachments. The backend normalizes all of that into a shared Part model, runs deterministic McMaster resolution, and returns a Project the React editor can fix by hand. There is no database in the MVP; projects live in an in-memory store until the process restarts.

That scope cap matters. The tool is not a general procurement system, not an ERP connector, and not an AI shopping assistant. It is a BOM hygiene layer for 3D-print projects where the author already listed hardware but did not link SKUs.

Keeping the problem correctly scoped

Feature creep arrived on day one: price quotes, cart checkout, multi-vendor comparison, automatic checkout on McMaster, LLM "smart matching" for vague lines like "some screws."

Each of those expands legal surface, needs credentials, or hides errors behind probabilistic guesses. The execution plan (PLAN.md) keeps phases explicit: ingest MakerWorld reliably, parse to structured parts, attach McMaster links with honest confidence, ship an editor + CSV export, then iterate with regression tests.

Deferred by design:

• Persistent projects / user accounts
• Automatic McMaster ordering
• Hosted third-party McMaster APIs that require external keys
• LLM-based description parsing

The product promise is narrower and more defensible: turn a MakerWorld BOM into an editable McMaster-oriented spreadsheet the human still verifies.

No LLM matching strategy

MakerWorld descriptions are messy, but they are not unstructured noise. They repeat patterns: 13 x M3-16 mm, assembly notes (Left module: 4 x M3-8), quantity prefixes, and section headers like "Material required."

The description parser is explicitly rule-based, no AI (parsers/makerworld/description.py). Extraction uses:

• Section keyword detection (BOM, Material required, fastener headings)
• Shared quantity regexes in parsers/helpers/bom_quantities.py
• Hardware signal checks (has_hardware_signal) to skip filament and printed parts
• Line-level parsing without model inference

Matching McMaster rows follows the same philosophy. Resolution is a tiered offline resolver (vendors/mcmaster/tiers.py): curated catalog JSON, length tables for common SKUs, embedded part numbers in text, filtered browse URLs with metric thread + length facets, then broader category or site search. Optional live enrichment (official B2B API or Playwright browse tables) sits behind feature flags and never replaces the offline path.

Why avoid LLMs here:

When a line is ambiguous, the UI shows low confidence and a search link, not a fabricated SKU. Humans remain the final matcher.

Enforcing bolt length (and diameter) rules

Catalog hits feel authoritative until a MakerWorld line says M3×16 mm and the matched SKU is a 20 mm cap screw with high heuristic score. Fasteners need post-match verification, not just string similarity.

hardware_spec.py extracts metric diameter and length from many prose shapes:

• Joined forms: M3x16, M3-16 mm, M3×16 mm
• Split forms: M4 screw 40 mm
• Trailing length: socket head screw 16 mm

primary_fastener_spec() prefers original_name over polluted specification fields (MakerWorld often stuffs assembly notes into the wrong column).

After a catalog match, hardware_match_verify.py compares BOM specs to catalog title specs:

For screws without length, the matcher prefers filtered browse URLs with thread-size~m3/ and length~16-mm/ path facets over a blind catalog SKU. Ranking deliberately elevates filtered browse (~0.90) above a weak catalog guess (~0.72–0.84) when metric thread and length are known.

That is the core bolt-length lesson: treat length as a first-class constraint, not a post-search filter the user discovers after clicking.

Tiered matching confidence

A single "confidence" number hides too much. The repo uses match tier + numeric score + editor status.

Offline tiers (McMaster adapter docs):

Heuristic scoring adds signals (hardware keywords, digits, spec field present), but catalog hits no longer auto-score 1.0. Verification can restore 1.0 on verified or corrected; mismatches clamp confidence.

resolve_match_status() maps tier + score to UI labels (likely, possible, unlikely). The editor highlights rows that need eyes before export.

Optional live paths are separate tiers:

• Official Product Information API (MCMASTER_API_ENABLED): B2B client cert, subscription limits, enriches descriptions and product status; credentials never reach the browser (API docs).
• Browse table resolve (MCMASTER_BROWSE_RESOLVE_ENABLED): Playwright intercept of ProdPageWebPart.aspx JSON; slow, integration-only.

Default dev and CI run offline tiers only, so tests stay deterministic.

Duplicate parts across BOM sources

MakerWorld often lists the same hardware twice: embedded BOM, description prose, and an uploaded spreadsheet. Without a merge policy, quantities double and McMaster matching runs twice on identical lines.

merge_parts() in parsers/helpers/parts_merge.py combines lists with a simple rule:

key = (part.original_name.strip().lower(), part.quantity)

First occurrence wins; later duplicates drop. That is deliberately conservative:

• Same name, same quantity → duplicate (skip)
• Same name, different quantity → kept as separate rows (may be intentional: sub-assemblies)
• Slightly different strings (M3x16 mm vs M3-16 mm socket head) → not deduplicated yet (normalization is a follow-up)

Open questions documented in the codebase:

• Should normalized fastener specs merge across wording variants?
• Should quantities sum when duplicates clearly refer to the same SKU?
• Should embedded BOM always beat description prose on conflict?

The current answer favors no silent quantity math until regression fixtures prove it safe. Wrong totals are worse than duplicate rows a human deletes in the editor.

Notebook-driven development

The repo treats Jupyter notebooks as the first implementation surface, not demo-only artifacts. Numbered notebooks mirror pipeline stages:

Rules from the notebook-driven pipeline skill:

1. Prototype in the stage notebook with fixtures in data/ or curated URLs in data/regression_urls.json
2. Promote stable functions into backend/services/
3. Re-import services from notebooks (no forked logic)
4. Add pytest coverage before exposing new behavior on FastAPI routes

notebook_utils.py adds Jupyter-only helpers (safe_scrape, safe_import_project, offline fallbacks). The website always runs the live pipeline; notebooks may cache when MakerWorld rate limits bite.

./scripts/dev.sh starts API, Vite, and JupyterLab together so the /notebooks page and import UI exercise the same code path.

API and terms-of-service considerations

Two external sites define the legal and operational envelope.

MakerWorld (scraping)

All MakerWorld HTTP traffic is server-side (httpx, optional Playwright). The browser never fetches model pages directly (CORS and logic exposure). Outbound politeness is enforced in rate_limit.py:

SCRAPER=auto tries httpx first, falls back to headless Chromium on 403/proxy failures. Regression URLs and fixtures reduce how often CI hammers live pages.

Operational stance: cache HTML for tests, throttle in dev, disable rate limits only locally (RATE_LIMIT_ENABLED=0), and return 200 with warnings when a project has no BOM instead of retry-storming ambiguous pages.

McMaster-Carr (linking, optional API, optional browse)

McMaster linking is mostly URL construction + curated catalog data, not bulk scraping in the default mode.

Constraints documented in-repo:

• Official API requires approved B2B account, client certificate, per-part subscription limits, and 24h tokens. Enabled only when credentials exist; responses stay server-side (Product Information API).
• Browse table resolution uses Playwright against public product tables. Docs note McMaster expects automated load stay tied to purchasing decisions: keep outbound intervals, disable browse in CI, prefer offline catalog + filtered browse URLs by default.
• Guardrail tests scan for credential leaks in API responses and tracked files.

Third-party hosted McMaster wrappers were avoided so the vendor adapter template stays self-contained and keyless for MVP users.

Stack snapshot

Full architecture: docs/architecture.md.

What shipped vs what is next

Phases 1–5 of the execution plan are marked complete: ingest, parse, match, editor, export. Phase 6 iteration continues (regression notebook, bulk edit, catalog integrity scripts).

Likely next hardening steps:

1. Smarter duplicate merge on normalized fastener specs (sum quantities when safe)
2. Cross-check API Specifications[] against hardware_spec extractions
3. Optional persistence without turning the MVP into a multi-tenant SaaS
4. More regression fixtures from real MakerWorld exports (data/regression_urls.json)

The repo exists because printable project pages already contain BOMs, but not purchasable links with honest uncertainty. The implementation bet is that deterministic parsing + tiered McMaster URLs + length verification beats an LLM that sounds confident while pointing at the wrong screw.

If that workflow matches a shop process, clone makemcmasterBOM, run ./scripts/dev.sh, and start in 04_match_mcmaster.ipynb before touching the matcher.