For your AI: ⬇ Download the raw file  ·  updated 2026-09-05: layering research + the full sequenced test plan

Other AIs Beyond SAM3 — Researched Shortlist (2026-09-05)

From: Skippy (Nate's AI) · For: Alex (and his AI), Nate How this was produced: a 6-angle web research sweep covering lane-detection/vector-map models, thin-structure segmentation, the post-SAM3 open-vocab foundation stack, VLM pointing/grounding, pose-model upgrades, and adjacent domains plus technique tricks. The sweep surfaced 44 raw candidates; we deduped and ranked them to 8, then a separate agent adversarially fact-checked every finalist by opening the actual repo, paper, and license (several fetched checkpoints to confirm they download). A final completeness critic reviewed the whole sweep. Claims below carry the verifier's corrections rather than the researcher's optimism.

The question (Alex, 2026-09-04): "im convinced we can get better accuracy with using some kind of Ai and is there any other than SAM3 we should be looking at?"

The one-paragraph answer

No better "find faint painted stripes" brain sits on a shelf. The only real SAM3 successor as of Sept 2026 is SAM 3.1, a video-speed release with no accuracy gains, and 2026 chatbot/vision AIs still fail at tracing thin lines: three 2026 papers show they lose the line and jump to a nearby look-alike, which is exactly our batten problem. The closest architectural match to our problem in all of computer vision is lane detection from self-driving cars (faint painted line on a surface in, ordered point sequence out), and one proven, permissively licensed model, CLRerNet, is worth fine-tuning as a challenger finder. The two highest-value moves cost almost nothing and use no new model at all: a pretrained straight-line detector to find the boom and veto battens/rigging, and a label audit that finds our own under-traced training labels — the root cause behind the missed faint and bottom stripes.

The verified shortlist (rank = recommended order, after critic re-rank)

1. Label audit: find our under-traced training labels

2. CLRerNet, the lane-detection challenger finder

3. DeepLSD / ScaleLSD: boom finder plus batten/rigging veto, zero training

4. Test-time augmentation over deep20 (multi-scale, bottom-crop, low threshold)

5. DINOv3 dense backbone: faint-stripe evidence heatmap (linear probe first)

6. clDice / Skeleton Recall loss (only if SAM3 stays in the production path)

7. MM-Grounding-DINO: open-vocab boxes for boom and batten classes

8. YOLO26-pose, the drop-in finder swap

What got dismissed, and why it matters

Cross-cutting rules from the critique (adopt for every test above)

  1. Holdout hygiene: carve a dev slice from training data for tuning thresholds and margins; touch the frozen 332 only for final gates; apply the 73-content-dupe exclusion (exclude-holdout-dupes.mjs) to every holdout diff.
  2. Cheap data levers the sweep under-weighted: synthetic faint-stripe degradation (fade well-labeled stripes to manufacture faint training examples at zero labeling cost) and batten/rigging hard negatives. Pseudo-labeling the ~10k unlabeled uploads is real but must wait until the label audit fixes the teacher, or it bakes in exactly our blind spot.
  3. The boom-labeling budget is one decision: DeepLSD scoring, MM-GDINO classes, and a pose boom-class all want the same one-time boom annotations. Decide the target once; Alex's Cape31 correction pass is the natural vehicle (docs/cape31-program-plan.md step 4).
  4. Already answered internally: the critic asked whether SAM3 full-image concept prompting was ever tried as a recall source. It was: hop-eval run D (SAIL-132) is exactly that, and it found 1039 junk lines vs 777 real. The pending idea is the curvature gate (stripes sag; battens and shrouds stay straight; SAIL-134), which is also what makes DeepLSD's veto principled.

Recommended sequence (cheap to expensive, each gated on the last)

  1. Label audit plus random control (script only); SAIL ticket to follow.
  2. TTA day on deep20 (no training; doubles as a test of the label-root-cause hypothesis).
  3. DeepLSD zero-shot boom/veto screen (needs the boom-label decision).
  4. Single-valuedness audit of the 332 labels, then the CLRerNet fine-tune (the challenger-finder bet).
  5. DINOv3 high-res linear probe (one pod-day).
  6. YOLO26 swap when a retrain is scheduled anyway.
  7. clDice/skeleton-recall only if SAM3's production role is reconfirmed; MM-GDINO fine-tune only after boom/batten boxes exist.

Every eval number quotes its id file and freeze date, per house rules. Nothing auto-promotes.

Added 2026-09-05: How experts combine multiple models ("layering")

Second research sweep, four angles: production cascades, ensembling, foundation-model composition, and geometric/non-model layers. It surfaced 27 patterns, and every worth-adding claim was adversarially fact-checked against its primary sources; the corrections from that verification are baked into what follows.

The six rules the field follows

  1. A refiner only works on inputs like the ones it trained on. Cascade R-CNN's core finding: running the same model twice on its own output adds nothing, and running a refiner on inputs better than its training band makes them worse. This single rule explains both of our past negative results, the keypoint model redrawing its own zoomed crops and the early SAM3 refine variant. Both are the documented failure mode of the pattern done wrong.
  2. Split recall and precision into different models. The medical-imaging CAD pattern: stage 1 deliberately over-proposes and a small stage-2 classifier kills the junk; the LUNA16 lung challenge institutionalized this with candidate sensitivity around 98% before filtering. Under this rule our SAM3-solo result (1039 junk vs 777 real) reads as a healthy over-sensitive stage 1 that is missing its stage 2.
  3. Judge before you redo. A verifier that only scores, accepts, or rejects can never make a good output worse, while an ungated redrawer can. Gates need their own calibration check, because miscalibrated confidence routes wrongly in both directions. Google's shipped example: MediaPipe hand tracking re-runs its detector only when landmark confidence drops.
  4. Downstream layers cannot add recall. The finder's proposal set is a hard ceiling. Recall for faint/bottom stripes and booms must come from an upstream over-proposer or from a geometric prior that says where to look, never from more refinement.
  5. Ensemble gains come only from diversity. Re-running the same model adds zero, which is why our YOLO-on-its-own-crops failed while YOLO+SAM3 agreement works. And models trained on the same labels share blind spots, so agreement can be confidently wrong about the same faint stripe.
  6. Keep the committee offline; ship one model. Netflix's $1M prize stack was never deployed, and Google distills ensembles into single models. Every proposed layer must show its marginal number on the frozen 332 with the stage bypassed, or it doesn't earn a place.

Validation: our pipeline is already the textbook stack

Added 2026-09-05: THE TEST PLAN — every candidate, sequenced

Scope: all 8 shortlist candidates above, 5 layering additions, the gaps the completeness critic found, Nate's escalation idea, and Alex's frontier-model question. Nothing here runs until Nate/Alex say go. Proposed discipline: max 2 experiments in flight at once; each posts its result to the chat when it finishes; every final number is scored on the frozen 332 (test-holdout-ids-canonical.json, frozen 2026-08-25) with the 73 content-dupes excluded; all threshold tuning happens on a dev slice carved from training data and never on the 332; no model promotes without a human call.

Phase 0: measurement only, no training, no builds (about this week, under $5)

# Test What it answers Effort
0.1 Per-stage scoreboard: end-metric on the 332 with each existing layer bypassed (finder alone / +rules / +SAM3 redraw) The ablation table every later item is judged against ½ day, CPU, existing envelopes
0.2 Label audit + blind random control (shortlist #1); the audit script is already running as of this morning Whether under-tracing is real and how widespread: top-100 worst-scored labels vs a random-100 base rate, human-eyeballed script done; one human hour
0.3 Confidence-to-error stratification: join per-stripe YOLO confidence with line error on the 332, plus SAM3-redraw delta per confidence band Whether confidence predicts quality (enables gates and a customer confidence badge), and where redraw helps or hurts ½ day, inference-only join
0.4 Curve-fit residual probe (layering): robust low-order fit per stripe (Huber or leave-one-out at our ~10-point count, where classic RANSAC is statistically thin), correlate residual with error, count batten-grab flags A free per-line quality score and a second gate signal. Today stripeMetrics' Catmull-Rom passes through every raw point, so one bad point bends the numbers we sell 1 afternoon, CPU
0.5 Candidate-pool recall check: are the faint/bottom stripes we currently miss present among SAM3-solo's 1,816 candidates? Whether the LUNA16 filter path can add recall or only cut junk; decides 1.3 ½ day, data on disk
0.6 Sail-outline registration probe (layering, sports-field pattern): outlines on the 332 via the sail-edge spike method; count missed stripes inside their predicted height band and known FPs outside the sail Whether one global sail fit can attack all three weaknesses. Template stays loose; sails are soft, unlike soccer pitches 1 day
0.7 Model soup of deep-slot-11/15/20 (same init, same data, same sampling: the verified-correct trio, whereas v26+deep20 have different objectives): shape-assert, uniform-average, one holdout eval; recalibrate BatchNorm before calling a negative Free accuracy at zero serving change ½ day, 1 eval
0.8 Single-valuedness audit of the 332 labels along the luff-leech axis CLRerNet's go/no-go (3.2); failure reroutes to a MapTR-style point-set head 2 hours
0.9 Escalation stats for Nate's rule ("only run the big model when fewer than 3 stripes are found"): how often production returns fewer than 3 stripes, and what solo SAM3 finds on exactly those photos Sizes the win of the escalation gate before building it. This is the industry's confidence-gated cascade pattern (NoScope, MediaPipe) ½ day

Phase 1: one-day builds, no retraining (gated on Phase 0)

# Test Gate to proceed Effort
1.1 TTA day over deep20 (2–3 scales, zoomed bottom-third crop, lowered threshold; manual passes since Ultralytics' TTA flag doesn't run for pose; no horizontal flip; custom polyline fusion). A null result counts as evidence for the label root cause proceed regardless; judge on recovered misses vs new batten FPs 1 day
1.2 DeepLSD zero-shot boom finder and straightness veto (collinear-merge long segments; curvature margin mandatory) needs boom truth on the 332, so Decision D1 1 day
1.3 Junk-vs-stripe crop classifier (LUNA16 stage 2): train on SAM3-solo's 1039 junk and 777 real (photo-level split, 332 untouched), measure ROC only if 0.5 shows the misses live in the pool; otherwise its value is FP reduction alone 1 day, minutes of GPU
1.4 Multi-prompt SAM3 redraw (the SAMRefiner idea: points, a loose box, and a mask prior mined from the polyline). Framed plainly as re-testing a closed negative with the one untested fix; first verify our SAM3 accepts a mask prior at all run after 0.3's per-band table exists 1 day plus GPU hours
1.5 Escalation gate v1: wire Nate's fewer-than-3-stripes rule (plus low-confidence routing if 0.3 shows signal) to trigger solo SAM3 plus the 1.3 filter only on flagged photos 0.9 sizes it; 1.3 supplies the filter 1–2 days

Phase 2: data work (gated on 0.2's audit results)

Phase 3: training bets (each is one run plus dual-ruler proof; gated on Phase 2)

Phase 4: system-level

Not doing, and why (one line each)

VLM line-tracing (three 2026 papers show them losing the line to a nearby look-alike, exactly battens); hosted APIs like DINO-X and Gemini trajectories (photos leave our infra and nothing can fine-tune); WBF-style averaging fusion (selection beats fusion while confidences are uncalibrated); a learned combiner (the Netflix lesson; our rules layer already plays that role, debuggably); deep ensembles in serving (artifact count, though the snapshot variant is fine offline for committee QC); top-down pose models (stripe crops overlap near-fatally); migrating to SAM 3.1 (a video-speed release).

Answers folded in

Decisions needed before the gated items