water-routefinder — development plan¶
Status note (implementation, 2026-09): the original design below (drafted while this repo still lived as
water-path/docs/water-pathfinder-plan.md) is preserved as historical context — most of it still holds. What actually shipped diverges in a few places the plan didn't anticipate:
- Naming: package
water_routefinder/ CLI-less, notwater_pathfinder— matches this repo's actual name and GitHub remote (FraSanvit/water-routefinder).- Delivery shape: a Snakemake workflow (
workflow/Snakefile), not atyperCLI. Thebuild/validate/inspectcommands in §5 below became Snakemake rules (workflow/rules/*.smk) driving the same underlying package functions.- Environment/tooling:
pixi(pixi.toml/pixi.lock), notuv/hatchling-only.hatchlingstill builds thewater_routefinderpackage itself; pixi wraps it with the runtime/dev environment and task runner (pixi run test,pixi run run-demo, ...).- Repository conventions: adopted from
modelblocks-org/data-module-template—resources/user/(input) vsresources/automatic/(downloads/intermediates) vsresults/(output),INTERFACE.yaml, aconfig/(user-editable, schema-validated) vsworkflow/internal/(non-user-editable settings) split. One deliberate departure from that template: it builds embeddable modules whoserule allrefuses to run standalone;water-routefinderis a standalone deliverable, sorule allactually builds the bundle(s).- Multi-provider downloads: ERA5 (§10's "keep configurable") shipped as a real provider (
sources/era5.py, wind + wave only — no ocean-current product), not just CMEMS, chosen independently per variable family inconfig/config.yaml.- Contract test deferred, not dropped:
test_contract_water_path.py(§7 M1, §8) is not yet written —water-pathitself still only reads the old v0.1 NetCDF format (itsio/environment.pyneeds a v0.2 Parquet port first; tracked as a follow-up in that repo, out of scope here). This repo's ownvalidate.py(+workflow/rules/validate.smk) is the real contract check for now, and there's no dependency on thewater-pathpackage (Network+ geometry helpers are vendored — seesrc/water_routefinder/network.py).- Added, not in the original plan: a per-network diagnostic plot (
results/{network}/{network}_diag_plot.png— route map, per-variable conditions over time, data-quality panel) and a dedicated input-format test suite (tests/test_input_format.py), both requested during implementation planning.- No offline mock provider:
sources/mock.py(§3, §7 M1'smocksource) was removed — for now the workflow supports onlycmemsandera5(a later stage may bring an offline provider back for demos/CI). This meansconfig/config.yaml's shipped example,pixi run run-demo, and the fulltests/integration/test_workflow.pyall need live CMEMS credentials now; the offline unit suite (pixi run test-unit) still runs without any, using a small test-only synthetic-dataset helper (tests/_synthetic.py, not part of the package or reachable from config) instead of a real provider.See
docs/contract.mdfor the frozen v0.2 bundle contract this repo targets, andCLAUDE.md/README.mdfor the as-built architecture and usage.
water-pathfinderis the sibling repository that builds the environmental-conditions bundlewater-pathconsumes. It scouts what lies along the route — winds, currents, waves — and handswater-patha ready-to-use table.This plan currently lives in the
water-pathrepo becausewater-pathfinderdoes not exist yet. On creation, move it towater-pathfinder/docs/roadmap.mdand keep only a one-line pointer here.
1. Purpose & scope¶
In scope
- Pull met-ocean source products (Copernicus Marine primary; ERA5 / Marine Institute optional).
- Harmonise them onto one standardised grid / cadence / convention set.
- Sample that field onto the route geometry of a water-path network.
- Emit the bundle defined by water-path docs/environment-data-format.md v0.2:
network/ (pass-through, possibly densified) + environment/conditions.parquet +
environment/conditions.meta.yaml.
Out of scope
- Any optimisation, physics, or vessel modelling — that is water-path.
- Serving grids to water-path (the harmonised grid stays internal; see ADR-0001).
Contract direction: water-pathfinder depends on water-path's published schema
(the spec doc + a JSON Schema), never on water-path code. water-path depends on neither.
2. Architecture — two stages¶
sources/* --fetch--> [ harmonise ] --regional grid--> [ sample ] --table--> bundle
(CMEMS, regrid + time-align + bilinear onto
ERA5, …) unify conventions + masks route vertices
harmonise— network-agnostic, independently testable. The reusable core (theatlite-cutout equivalent). Output: an in-memoryxarray.Datasetregional grid, standard variable names & units.sample— takes the grid + awater-pathNetwork; may densify route geometry to a target vertex spacing; interpolates to every route vertex at every timestep; computess_along_m; returns(table: DataFrame, network: Network)(network possibly densified).
3. Package layout (original plan — superseded by the as-built layout in README.md / CLAUDE.md)¶
src/water_pathfinder/
config.py # pydantic: sources, time range, densify target, output options
schema.py # the v0.2 contract as code — column set, dtypes, units, sidecar model
# (mirrors water-path; verified against its JSON Schema in CI)
sources/
base.py # SourceProvider ABC: fetch(bbox, start, end) -> xr.Dataset
mock.py # deterministic synthetic field, offline — default for tests & demo
cmems_phy.py # surface currents (uo, vo) via `copernicusmarine`
cmems_wav.py # waves (VHM0, VMDR, VTM10) via `copernicusmarine`
cmems_wind.py # 10 m wind (eastward_wind, northward_wind)
era5.py # optional fallback via `cdsapi`
harmonise.py # regrid to common grid + resample time + convention/unit unification
sample.py # densify geometry, bilinear grid->vertices, along-track distance
bundle.py # assemble & write network/ + conditions.parquet + conditions.meta.yaml
provenance.py # source dataset ids + versions + access time; network_ref hashing
validate.py # check a bundle against schema.py (reused by the CLI and CI)
viz.py # `inspect` quicklook: ranges, gaps, per-route field plots
cli.py # build / validate / inspect
tests/
test_schema.py test_sample.py test_harmonise.py test_bundle.py
test_sources_mock.py
test_cmems_integration.py # @pytest.mark.integration, needs credentials, skipped in CI
test_contract_water_path.py # installs water-path, runs open_conditions() on our output
docs/
examples/dublin-bay/ # network + a committed demo bundle (mock source)
4. Key interfaces¶
class SourceProvider(ABC):
"""One met-ocean product family, returned on its native grid with standard names/units."""
name: str
provides: set[str] # e.g. {"current_speed", "current_to_direction"}
def fetch(self, bbox: BBox, start: datetime, end: datetime) -> xr.Dataset: ...
def harmonise(datasets: list[xr.Dataset], *, target_step: str = "1h") -> xr.Dataset: ...
def sample(
grid: xr.Dataset, network: Network, *, densify_km: float | None
) -> tuple[pandas.DataFrame, Network]: ...
def write_bundle(out_dir: Path, table: DataFrame, network: Network, meta: SidecarMeta) -> None: ...
BBox is derived from the network's route geometry plus a one-cell margin.
5. CLI (original plan — shipped as a Snakemake workflow instead; see README.md)¶
water-pathfinder build NETWORK_DIR --start DATE --end DATE
[--config cfg.yaml] [--out DIR] [--densify-km 1.5]
[--source cmems|era5|mock] # default: from config
water-pathfinder validate BUNDLE_DIR # exits non-zero on any schema violation
water-pathfinder inspect BUNDLE_DIR # human summary + quicklook plots
Config file (YAML):
sources:
current: { dataset_id: "cmems_mod_glo_phy_anfc_0.083deg_PT1H-m", variables: [uo, vo] }
wave: { dataset_id: "cmems_mod_glo_wav_anfc_0.083deg_PT3H-i", variables: [VHM0, VMDR, VTM10] }
wind: { dataset_id: "cmems_obs-wind_glo_phy_my_l4_...", variables: [eastward_wind, northward_wind] }
time: { start: 2024-01-01, end: 2024-12-31 }
sample: { densify_km: 1.5 }
6. Dependencies¶
| need | choice | note |
|---|---|---|
| CMEMS access | copernicusmarine |
official client; token auth |
| ERA5 (optional) | cdsapi |
behind an extra |
| arrays / IO | xarray, numpy, netcdf4 (or h5netcdf) |
internal grid handling |
| regridding | xarray.Dataset.interp (bilinear) to start |
avoids the xesmf/ESMF conda dependency; revisit if conservative regridding is needed |
| geometry | pyproj (geodesic densification + distance) |
water-path uses haversine; pyproj is more accurate for this step |
| table out | pyarrow |
Parquet |
| config / sidecar | pydantic, pyyaml |
|
| CLI | typer (or click) |
superseded — Snakemake is the interface |
| build | hatchling, src/ layout |
mirror water-path; wrapped by pixi |
7. Milestones¶
| # | Deliverable | Exit criterion |
|---|---|---|
| M0 | Repo skeleton, CI, schema.py, validate.py, config.py |
validate runs; schema round-trips; CI green |
| M1 | mock source + harmonise + sample + bundle + build CLI |
water-pathfinder build examples/dublin-bay --source mock produces a bundle that passes water-path's open_conditions() (contract test). Replaces water-path's make_demo_conditions.py. |
| M2 | Real CMEMS sources (phy / wav / wind), auth, bbox-from-network, response caching | integration test builds a 3-day Dublin-Bay bundle from live CMEMS (credential-gated) |
| M3 | Harmonisation robustness: common grid, time resample, land-mask & gap policy, unit assertions, NaN policy | property tests on synthetic multi-grid inputs; documented policies |
| M4 | Geometry: densify-to-spacing, along-track distance, re-emit routes.geojson; degenerate-route handling |
sample tests incl. short / 2-vertex / already-dense routes |
| M5 | Provenance & reproducibility: full meta.yaml, network_ref hashes, deterministic output, --dry-run |
byte-identical re-run; provenance lists dataset ids + versions + access date |
| M6 | Docs + first tagged release | README, usage guide, contract reference, examples/dublin-bay; version policy tied to waterpath_env_schema |
As-built status: M0–M2 are implemented (mock + CMEMS + ERA5 sources, Snakemake build/
validate/diagnostics rules, resources/user/dublin-bay example, unit + input-format +
end-to-end workflow tests, credential-gated integration tests). M3–M6 remain open — see
"Out of scope / follow-ups" in the implementation plan history, or open an issue.
8. Testing strategy¶
- Default suite is offline:
mocksource, synthetic grids, committed golden bundle (tests/integration/expected/). - Contract test (
test_contract_water_path.py): not yet added —water-pathneeds a v0.2 reader first (see the status note at the top of this file).validate_bundle()+workflow/rules/validate.smkare the real contract check today. - Integration tests (
-m integration): real CMEMS/ERA5, gated onCMEMS_USERNAME/_PASSWORDor CDS credentials; run nightly and on demand, not on every PR. - Golden-file: small committed bundle regenerated and diffed; drift is a failing test
(
tests/integration/test_workflow.py). - Schema sync: not yet added (no published
water-pathJSON Schema yet) — left as aTODOin.github/workflows/ci.yml.
9. Versioning & compatibility¶
water-pathfinderwrites thewaterpath_env_schemavalue it targets into every sidecar.water-pathgates on the major of that value.- A breaking schema change = coordinated major bump: new
water-pathdocs/environment-data-format.mdversion + newwater-pathfinderminor that targets it + an ADR inwater-path. water-pathfinder's own version is independent (feature/bugfix cadence).
10. Open questions¶
- Regridding:
xarray.interpbilinear is enough for smooth met-ocean fields at these scales; confirm before M3. Conservative regridding (xesmf) only if fluxes matter. - Wind source: CMEMS blended L4 vs ERA5 vs scatterometer — kept configurable per family in
config/config.yaml(shipped); the Dublin-Bay example defaults tomock. - Cross-product time resolution: waves are often 3-hourly, currents hourly — resample all to one step (config, default 1 h) and document the interpolation.
- Wave parameters: v0.2 ships
wave_period(mean, Tm) only. Revisit once the physics engine specifies its added-resistance model (may want Tp or directional spread). - Shared schema package: JSON Schema fetched in CI is the lightweight option now. If a
third consumer appears, promote it to a tiny
water-path-env-schemapackage both import.
11. First steps on repo creation (original plan — see the status note above for what actually happened)¶
hatch new water-pathfinder(or copywater-path'spyproject.tomlskeleton),src/layout.- Port
water-path'sexamples/demo/environment/make_demo_conditions.pylogic intosources/mock.py(it already generates a plausible synthetic field). - Implement
schema.pyfromdocs/environment-data-format.mdv0.2 +validate.py. - Wire
harmonise(pass-through for a single mock source) →sample→bundle→buildCLI. - Add the
water-pathcontract test. Green M1. - Delete
water-path'smake_demo_conditions.py; regenerate the demo bundle withwater-pathfinder build --source mockand commit it intowater-path.
Not yet done (tracked as follow-ups, in water-path unless noted): step 5 (contract test —
blocked on water-path's v0.2 reader) and step 6 (touches the other repo).