Can a simple classifier tell real stripes from junk in the raw candidate pool?

Promising but not ready to ship. AUC 0.89 — a good signal. At a setting that keeps 90% of real stripes, it cuts total misses by about 33%, but costs roughly 1 extra junk line per photo.

What it means for SailScan

Test 0.5 showed about half of our misses are sitting in SAM3's raw candidate pool, waiting to be filtered out from the junk. This classifier is a first attempt at that filter — it works, reaching about two-thirds of that recoverable ceiling, but at a real junk cost (1 extra wrong line per photo on average). Worth prototyping further with more features before shipping; not a drop-in win yet.

0.89

classifier AUC

33%

of all misses recovered

1.0

junk lines / photo cost

The detail

Measure	Value
Classifier accuracy (AUC, gradient boosting)	0.8947
Real stripes kept at the chosen threshold	90.0% (699 of 777)
Junk lines let through at that threshold	31.8% (330 of 1039)
Of test 0.5's 58 total misses, how many does this classifier recover?	32.8% (19 stripes)
Junk lines added per photo	1.0

A small classifier trained only on line shape (no pixels) to guess "real stripe" vs "junk" among SAM3-solo's raw candidates.

Source: SAM3-solo candidate pool, 330 of the 332-photo holdout (2 photos had no candidates). 5-fold cross-validated.

raw data (JSON) →