# Independent algorithm comparison protocol

## What this comparison can establish

This experiment measures **schedule correctness, supported itinerary scope, and consistency of the outputs**. It does not establish clinical accuracy, reduced jet lag, a person's actual circadian phase, or medical superiority. No measured traveller sleep, dim-light melatonin onset, light exposure, adherence, or clinical outcomes were supplied. Agreement with another simulator is a model comparison, not physiological ground truth.

The selection rule is deliberately not a single weighted accuracy percentage:

1. Confirm the upstream algorithm runs unmodified, record commit/license/runtime, and preserve its raw output. An adapter may translate types and timestamps; it must not quietly repair the schedule it is judging.
2. Check supported scope before judging results. A library that takes only an offset or a light trace is not a working itinerary planner. A single-leg engine run separately on every leg is not a valid multi-leg result. An unsupported empty result cannot win by containing no contradictions.
3. Apply independent physical and publication invariants. An algorithm with unresolved ownership violations is unsuitable for shipping as a complete replacement even if its phase model is more sophisticated.
4. Inspect coverage and feasible routine context. Suppressing every action trivially eliminates action conflicts but is not a usable plan.
5. Compare physiology assumptions and parameter mapping separately. A replacement that passes all engineering checks still needs empirical evidence before claiming superior clinical accuracy.
6. Repeat the frozen battery after integration, and run the reconstructed reviewer regression separately in the app. The known reviewer case is a regression holdout, not an unseen scientific validation sample.

## Frozen inputs

`fixtures.json` contains 30 synthetic plausible journeys, not claims about real scheduled airline services: ten singles, ten round trips, and ten multi-leg journeys. Five multi-leg trips finish at home and five end elsewhere. There are 61 primary flight legs; the separate reviewer holdout adds two. Airport-local timestamps are converted to UTC using the named IANA zone at each event, independently of any planner. No main-corpus trip has an unknown surface transfer or ambiguous DST clock input.

Shared traveller: habitual sleep 23:00–07:00, three preparation days, comfortable pace, caffeine and naps enabled, supplements disabled. The reference instant is exactly three elapsed days before the first departure. `finalRecoveryDays: 7` describes the requested output horizon, **not a physical departure**. A full final-stop sleep that crosses the report horizon is not a trip conflict.

### The planners do not have identical constraint sets

“Comfortable” is not a standardized intervention dose. SunShift's `SleepShiftPreferences` limits ordinary nightly prescription changes to one hour and bounds the schedule two hours from its sleep anchor; explicit user limits can tighten those bounds. Dawnward's nearest preset, “balanced,” selects a nominal one-hour daily advance or 1.5-hour delay. That rate is not an equivalent two-hour maximum displacement from the person's habitual routine. Its output can ask for more disruptive bedtimes while SunShift deliberately accepts residual estimated phase error. Neither engine's predicted daily phase movement is measured adaptation.

Consequently, a smaller residual phase error or faster simulated convergence cannot establish a better plan unless sleep opportunity, acceptable schedule displacement, feasible light exposure and user constraints are comparable. The independent ownership tests use the common hard facts both algorithms received. They do not reward a more disruptive prescription, and the scientific light-trace simulations must be reported separately as conditional model estimates. A circadian simulator supplied with an assumed light trace is not itself an optimizer satisfying SunShift's sleep constraints.

Dawnward also has no equivalent meal planner or configurable seven-day maintenance horizon. “No meal conflict” is **unassessable**, not a feature win, when an engine supplies no meals. Likewise, source completion estimates cannot substitute for visible routine guidance on the remaining requested days. External engines are judged only on features they actually emit; absent required product features are listed as integration gaps rather than favorable zeros.

Primary scenarios were frozen before the integration results were scored. Their canonical sorted compact JSON SHA256 is `c928a136ea55ba4cdf9402a882070477cdb95c6bd8e46504ea80578fbfbf8ee2`. The later correction of H01's recorded preparation cap to four days did not change any primary scenario. Full-file SHA256 after that correction is `586097364a4b42fdb8e91f5352e20fd9a2d78706955cb6b9113f50737622f9ac`.

| IDs | Journeys | Main stressors |
|---|---|---|
| S01–S02 | SFO→LHR; LHR→JFK | East/west, overnight/daytime |
| S03–S04 | LAX→NRT; NRT→LAX | Date line both directions, earlier local arrival |
| S05–S07 | JFK→DEL; DEL→KTM; SYD→ADL | Half-hour and quarter-hour zones; 15- and 30-minute changes |
| S08–S10 | SIN→HND; AKL→HNL; GRU→LIS | Near midnight, previous-date arrival, European DST during flight |
| R01–R02 | SFO↔CDG; JFK↔LHR | Ten-day vs 36-hour stay, competing adaptation targets |
| R03–R05 | SYD↔LAX; DXB↔DEL; LHR↔JFK | Date line, 16.5-hour stay, both European and U.S. DST changes |
| R06–R08 | SIN↔AKL; YYZ↔HND; HNL↔AKL | Early and midnight returns, repeated/skipped dates |
| R09–R10 | CDG↔DXB; LHR↔SIN | 06:00 vs 23:30 return departures |
| M01 | JFK→LHR→SIN→LHR→JFK | Two-hour and 2.5-hour hub connections |
| M02 | LAX→HNL→AKL→SYD→LAX | Pacific circuit, multiple short stays |
| M03 | JFK→LHR→DXB→SIN→JFK | Rapid target changes, 18h35 final flight |
| M04 | DEL→KTM→BKK→DEL | Quarter-hour and half-hour zones |
| M05 | SFO→HND→SIN→HND→SFO | 17-hour overnight and three-hour connections |
| M06 | LHR→JFK→LAX | Open journey, European DST during stay, domestic onward leg |
| M07 | SYD→ADL→PER→SIN | Australian DST, half-hour zone, non-DST regions |
| M08 | AKL→HNL→LAX | Open journey, previous-date arrival, late onward flight |
| M09 | CDG→DXB→DEL | Open all-connection journey, 7h15 overnight hub stay |
| M10 | JFK→GRU→LIS→LHR | Open journey, European DST, southern/northern hemispheres |

H01 reproduces the reviewer itinerary: JFK September 29 19:00 to LHR September 30 07:00; LHR October 7 12:00 to JFK 15:00, all in 2026 and airport-local. Its recorded preparation cap was four days, represented by `profileOverrides`. Its reference instant remains the declared synthetic three-day UTC lead because the original app-run timestamp was not captured. Do not describe this as an exact reconstruction of every original hidden setting. The main 30 use the shared three-day preparation cap.

`validate_fixtures.py` independently checks every local/UTC round trip, positive elapsed duration, airport and zone continuity, at least one hour between consecutive flights, category counts and the five-open/five-closed split. It rejects ambiguous local input instead of silently choosing an occurrence. Flight durations span 1h30–18h35 and intermediate ground stays span two hours–291 hours.

## Normalized outputs

`evaluate_schedules.py` reads JSON arrays, `{results:[...]}`, or `ALGORITHM_COMPARISON` log lines. Each record includes scenario ID, system and version, support status/reason, normalized events, optional normalized day/phase context, and raw source output. Preserve source-local labels and explicit source timing alongside normalized UTC. For each event distinguish `durationSource: point_marker`, an explicit interval/window, a resolved full sleep episode, or a sleep interval inferred from the source's next explicit wake marker.

In particular, a fifteen-minute UI expiry on a bedtime or meal marker is **not** its required activity duration. A wake marker at the exact flight arrival boundary is valid. A flexible light window may overlap sleep in part when a sufficiently long contiguous awake opportunity remains. A nap choice window does not imply the person must sleep for the whole window.

Events displayed by the app (`visible`) are evaluated separately from events the resolver accepts (`eligibility: eligible`). Rejected future candidates must not appear as ordinary instructions. `suppressed`, `historyOnly`, `conflict` and `preview` are not accepted. The confirmed-future fixture set is used for this publication check; this does not assert that a historical completed action can never be shown in a history interface.

## Independent measures

| Measure | Oracle and treatment |
|---|---|
| Physical ownership | Origin ground ends at actual departure; flight owns `[departure, arrival)`; destination ground ends at the next actual departure or remains unbounded at the final stop. Full published intervals and publication expiry must fit; point starts must fit. |
| Invalid intervals | Negative end times, nonpositive actual windows, and nonpositive publication validity are separate findings. Multiple findings on the same event are not multiple independent failures. |
| Publication consistency | Visible unaccepted candidates are counted separately from valid accepted events. This isolates the reviewer-class defect from the physiology engine. |
| Sleep consistency | Full or explicitly inferred sleep episodes cannot overlap. Required awake point instructions cannot fall strictly inside an accepted sleep episode. Awake windows wholly inside sleep are impossible; flight wake and sleep windows cannot overlap. App-exported quiet-card sleep copy is audited as a published instruction. Inferred Dawnward sleep intervals are annotated, not presented as explicit full-interval source declarations. |
| Caffeine consistency | Explicit caffeine-allowed and caffeine-cutoff windows cannot overlap. The evaluator does not invent a cutoff interval from a source's point marker. |
| Light consistency | Simultaneous opposing light windows are contradictory. Seek-light requires a contiguous awake opportunity at least as long as its explicit required session; without a declared session length only wholly sleeping through the window is a hard finding. |
| Duplicate episodes | Identical type/purpose/episode/start duplicates are flagged as presentation advisories. Exact repeated same-leg sleep targets are reported as duplicate sleep instructions, not incompatible overlapping sleeps. Repeated labels at different times are not automatically duplicates. |
| Quiet-day coverage | Every complete civil day physically spent at a stop needs visible context. Arrival/departure partial dates are excluded from this conservative measure. Final-stop checks use complete dates within the requested horizon. An action-free maintenance day may pass without adding interventions. |
| Phase continuity | When consecutive normalized day intervals touch, carried body-phase end/start estimates must agree within 0.05 hours modulo 24. Missing phase records are unassessable. This checks continuity, not the biological correctness of the estimate. |
| Capability | Missing/unimplemented itinerary support, generation failure and missing scenario outputs remain explicit. An algorithm without day records receives “unassessable” coverage rather than zero missing days. |

These measures do not audit airport transfer/check-in buffers because the common fixture schema supplies no comparable transfer constraints. They do not infer supplement timing, caffeine pharmacokinetics, hotel availability, sleep quality or real-world adherence. Action-copy correctness beyond exported quiet-card sleep still requires source-specific inspection. Flight cabin light exposure is an assumption, not measured data.

Seventeen evaluator self-tests cover these boundaries and the newer caffeine, nap, flight wake/rest, publication-expiry and quiet-card checks. The separate fault-injection script then applies 218 negative controls across the 31 actual app exports; see [the direct-conflict audit](STRICT-CONFLICT-AUDIT.md).

## Reproducible commands

From the repository root:

```sh
python3 validation/algorithm-comparison/validate_fixtures.py
python3 validation/algorithm-comparison/evaluate_schedules.py --self-test
python3 validation/algorithm-comparison/evaluate_schedules.py \
  validation/algorithm-comparison/results/sunshift-before.json \
  validation/algorithm-comparison/results/sunshift-after.json \
  validation/algorithm-comparison/dawnward/results.json \
  validation/algorithm-comparison/references/jetlag-results.json \
  --output validation/algorithm-comparison/results/evaluation.json
python3 validation/algorithm-comparison/compare_stability.py \
  validation/algorithm-comparison/results/sunshift-before.json \
  validation/algorithm-comparison/results/sunshift-after.json \
  --output validation/algorithm-comparison/results/before-after-stability.json \
  --require-identical
```

The last command verifies accepted recommendation signatures, including required session duration, and body-phase day traces, with no reference to publication scores. This ensures a publication fix cannot claim improvement merely by removing valid recommendations. The project workspace is one directory above that repository; prepend `Sunshift/` when running there. Source snapshots, runtime instructions, adapter parameter mappings and source-specific limitations accompany each external runner. Re-run the same commands after updating any source or fixture and retain the old raw result files for a before/after comparison.