Skip to content

FIELD EVIDENCE / PART II · OC01 AUDIT

Every claim.Back to evidence.

We re-tested our own corrections against NOAA’s MarineCadastre AIS archive. Second-level vessel positions confirmed the passage — and forced us to revise what proximity alone could prove.

Open the auditRead Part I
OBSERVATION → CORROBORATION / CONCEPT
464,413
AIS pings in 100×100 km box, 48 h
809
Unique vessels broadcasting at OC01
6.81 km
JOSCO HUIZHOU CPA · 2019-03-09 12:39:55 UTC
0 / 800
Chunks without a vessel within 10 km

THREE AUDITS, ONE TRUTH

Every method we tried surfaced its own bias.

01

v7.6 self-audit

We ran v7.6 (per-site calibrated, 96.4% honest test accuracy) over the OC01 corpus to verify the relabel. It agreed with every single correction. Then we noticed: v7.6 was trained on those exact corrected labels. The agreement is memorisation, not verification. We threw the result out.

560 / 560
Circular
02

Independent acoustic features

No neural network. Spectral flatness, blade-band energy (5-50 Hz), engine-band energy (50-500 Hz) on every chunk. The 560 "ship" chunks looked QUIETER than the 240 "ambient" chunks on every feature — wrong direction. We thought the corrections were wrong. They weren't. The 4 recordings have different absolute-dB baselines, and the "ambient control" recording itself contains vessels. Pure spectral comparison across recordings is unreliable.

d = -1.24
Confused
03

MarineCadastre AIS archive

NOAA Office for Coastal Management publishes second-level AIS positions for US waters. 464,413 pings over 48 hours within 100 km of OC01. We re-computed every chunk's nearest vessel within ±2 minutes of the chunk timestamp. Zero chunks have a vessel further than 10 km. OC01 sits at the western mouth of Juan de Fuca strait — the busiest shipping corridor on the US Pacific coast. "Ambient" at this site is a labeling convention, not an acoustic ground truth.

0 / 800
Decisive

PART I, CONFIRMED TO THE SECOND

JOSCO HUIZHOU was at 6.81 km. 12:39:55 UTC. 2019-03-09.

Part I claimed JOSCO HUIZHOU was within 10 km at 12:00 UTC, based on a coarse GFW daily query. NOAA's MarineCadastre archive has 698 AIS pings of the same vessel that day, accurate to the second. The closest pass was 6.81 km, at 12:39:55 UTC — inside the CNN flag window (12:14–12:44 UTC). The original case study was right.

JOSCO HUIZHOU track on 2019-03-09

Source: MarineCadastre.gov AIS archive · MMSI 477133400 · HKG-flagged cargo · 698 position pings

THE FINDING

Every chunk at OC01 has a vessel within 10 km.

All 800 chunks. Both labels. Every minute of the 48-hour audit window. The "ambient control" recording was selected to be quiet relative to known vessel passages, not relative to absolute silence. Absolute silence doesn't exist here.

Per-chunk nearest vessel distance over time

CPA DISTRIBUTION

Distance, not category.

Binary ship/ambient labels at this site lose information. The right encoding is a CPA distance band per chunk, attached to the named vessel that produced it.

Most chunks land between 6 and 10 km. The current "ship" labels overlap with "ambient" in this distance range — both contain audible passages.

5 – 7 km144 ship · 20 ambient
7 – 10 km416 ship · 220 ambient
> 10 km0 ship · 0 ambient
CPA distance histogram

WHAT THE MODEL ATTENDS TO — AND WHAT IT DOESN'T

Three pathways. One vessel. None depends on the others.

Grad-CAM on v7.6's last convolutional block. The CNN attends primarily to cavitation broadband (500–1000 Hz) and the lower engine band (80–250 Hz). It has no access to the blade-rate band (5–50 Hz)— preprocessing masks it before the model sees the spectrogram, by design, to prevent the model learning hydrophone-specific low-frequency noise.

Part I's case study found the JOSCO HUIZHOU signature in the blade-rate band via Welch's PSD on raw audio. The CNN cannot see that band. Yet both methods, plus MarineCadastre AIS positioning, converge on the same vessel. Three independent signal pathways — acoustic broadband, acoustic narrowband, and radio position — agreeing on ground truth. That's how robust detection systems are built.

Grad-CAM saliency for three oc01 chunks
A · JOSCO HUIZHOU window
CNN: ship · prob 0.581, attending to cavitation band and lower engine harmonics. PSD (Part I): +13 dB excess at 28–37 Hz (blade-rate). AIS: JOSCO HUIZHOU at 6.81 km, same minute. Three different bands, same vessel.
B · Mislabeled ambient
CNN: not_ship · prob 0.051. Attention pattern resembles the clean ambient on the right — the model inherited the dataset's label error because it was trained on it. AIS independently says WIND SONG / BALOS within 7 km. This is why cross-modal verification matters: the model alone could not detect this mistake.
C · Genuine ambient
CNN: not_ship · prob 0.054, diffuse attention. AIS: nearest vessel > 25 km. Both methods agree. The heatmap shows what "no concentrated vessel evidence" looks like in this model's representation.
Honest framing:We expected the model to attend to engine and blade-rate bands the way a human acoustician would point to them in a PSD plot. It doesn't. Cavitation broadband is a well-known vessel signature (Ross 1976, Urick 1983) — the model is using real physical features, just not the obvious narrowband ones. We caught this only because we ran the saliency analysis. Most teams don't. Documented in LIMITATIONS.

Methodology · open source

Reproducible end to end.

01

GFW vessel-presence

Hourly bbox sweep at 1/2/3/5/7/10/15/25/50 km. Brackets each vessel CPA upper bound per hour, enough to identify which named vessels were near the hydrophone and when.

9 radii
artifact
02

MarineCadastre raw AIS

NOAA Office for Coastal Management public archive. Filter to bbox, compute distance per AIS message, then find the per-chunk nearest vessel within +/- 2 minutes.

698 pings
artifact
03

Acoustic features

128-bin mel spectrograms, blade-band, engine-band, and spectral flatness. No neural network, which is why the cross-recording calibration drift is visible.

no CNN
artifact
OUT

The reproducible artifacts are per-chunk CPA, named-vessel attribution, and acoustic-feature JSON. Together they make each label correction traceable back to a timestamped vessel position and a measured signal profile.

One caveat for anyone reproducing this. The OC01 corpus holds 800 rows but only 400 distinct chunks: every chunk is present twice. Per-chunk results and group means are unaffected, but the group-level p-values in audit 02 were computed on the row count, so they are stronger than the underlying sample supports. That audit was discarded for the confounding described above, and none of the conclusions on this page rest on it.

METHODOLOGY CONTRIBUTION

What we're offering NOAA SanctSound.

Not "we found 560 corrections." We started with that claim and walked it back as the data demanded. What we have instead is sharper — and probably more useful for the next archive revision.

What we're offering

  • Per-chunk CPA + named-vessel attribution for the OC01 corpus we trained on, from MarineCadastre.gov
  • JOSCO HUIZHOU CPA 6.81 km @ 2019-03-09T12:39:55Z — confirming Part I to the second
  • Open-source audit scripts: GFW bbox sweep, MarineCadastre track filter, acoustic feature analysis
  • Methodology recommendation: binary ship / ambient labels lose information at high-traffic sites. Replace with distance band + dominant-vessel-class per chunk.
  • Extension to OC02–04 and any SanctSound deployment in US waters — same code, different bbox.