Architecture¶
What this is¶
The sibling repository that builds the environmental-conditions bundle
water-path consumes: it reads a route network
(harbours + routes.geojson), downloads the met-ocean products that cover it, harmonises them
onto one grid/cadence, samples that grid onto the route geometry, and writes the bundle
water-path's docs/environment-data-format.md v0.2 defines. water-path never sees a grid —
see its docs/adr/0001-environment-conditions-format.md for why.
No optimisation, physics, or vessel/consumption modelling here — that's water-path's job.
This repo does not depend on the water-path package (unpublished; its own reader still reads
the older v0.1 NetCDF format) — src/water_routefinder/network.py + io.py vendor a minimal
Network model and loader, kept in sync by hand with water-path's schemas/network.py /
io/routes.py.
The pipeline¶
resources/user/{network}/{routes.geojson,harbours.csv} <- your input
|
make_bbox -> download (cmems|era5, per family) -> harmonise -> sample -> bundle -> validate
| | |
| | +-> diagnostics
| +------------------------> availability
|
results/{network}/{network,environment,validation.txt,{network}_diag_plot.png,{network}_availability_map.png}
diagnostics and availability are two independent leaves, not a chain: diagnostics reads the
finished bundle (route-sampled values); availability reads the harmonised grid directly,
skipping sample/bundle, so it can show the source data's own spatial coverage rather than
what got interpolated onto the route.
Snakemake (workflow/Snakefile + workflow/rules/*.smk) orchestrates; the actual logic lives in
the plain, unit-tested src/water_routefinder/ package — workflow/scripts/*.py are thin
adapters that unpack snakemake.input/output/params/wildcards and call the package. That keeps
the core testable without invoking Snakemake, while still getting its DAG/caching/parallelism for
orchestration — the real win: CMEMS downloads are slow and credential-gated, so re-running
harmonise/sample while iterating doesn't re-hit the network.
Directory conventions follow
modelblocks-org/data-module-template:
resources/user/ (your input) vs resources/automatic/ (downloads/intermediates, safe to delete)
vs results/ (output); config/ (user-editable, schema-validated) vs workflow/internal/
(non-user-editable settings — provider capability matrix, default dataset ids). One departure from
that template: it builds embeddable modules whose rule all refuses to run standalone;
water-routefinder is a standalone deliverable, so rule all actually builds the bundle(s). This
workflow's structure is built on that template's rationale, and on the Snakemake data-module
practice of irm-codebase.
The bundle contract¶
The v0.2 contract (bundle layout, column schema, sidecar shape, direction
conventions) is water_routefinder/schema.py in code and contract.md in prose (a
frozen copy of water-path's docs/environment-data-format.md). validate.py (+
workflow/rules/validate.smk) is the real check today — there's no water-path-side contract
test yet, since water-path's own reader hasn't been ported to v0.2 (see the
roadmap's status note).
Network validation¶
Every route's path must start and end close to its origin/destination harbour: network.py's
PATH_ENDPOINT_TOLERANCE_KM = 2.0 — Network._cross_check() rejects the whole network otherwise,
with a clear error naming the offending route and the actual gap in km. Harbour ids referenced by
a route but not defined, duplicate harbour/route ids, and malformed GeoJSON/CSV are all rejected
the same way, at load time, before anything is fetched.
Basemap¶
The diagnostic plot's route-map panel draws real coastlines from a bundled, offline
Natural Earth 1:10m land extract
(resources/basemap/europe_land_10m.parquet, clipped to Europe + the North Atlantic approaches) —
not a live web-tile fetch. That was the first design (via contextily), and it turned out to be a
bad trade for a diagnostic plot: checked directly against three tile providers, fetch times ranged
from acceptable to tens of seconds per tile depending on network conditions, one provider now
requires an API key everywhere (including its formerly-free endpoint), and another blocks
unidentified automated clients outright. A bundled static file has none of that — no network call,
no timeout, no rate limit, no ToS, and it's faster besides. Set diagnostics.basemap: false in
config/config.yaml for a bare-axes plot instead.
Dependencies & sources¶
Every source provider (sources/cmems.py, sources/era5.py — the only two supported for now; see
config.PROVIDER_CAPABILITIES) returns data already in the v0.2 standard names/units/conventions —
the per-provider transform lives with the provider, not downstream in harmonise/sample. There
is no offline mock provider (removed; may return in a later stage for demos/CI that don't need
live credentials).
Conventions¶
src/layout, hatchling build backend, wrapped by pixi (pixi.toml— the canonical environment/task definition;pyproject.tomlkeeps the package pip-installable standalone). Import packagewater_routefinder.- pydantic for config/input validation (
config.py,network.py,schema.py). xarray.Datasetis the interchange format for met-ocean grids;pandas.DataFramefor the sampled table.tests/unit/mirrorssrc/water_routefinder/roughly 1:1, using a test-only synthetic dataset builder (tests/_synthetic.py— not part of the package or reachable from config) soharmonise/sample/bundlestay unit-testable without live credentials.tests/test_input_format.pytests network-file shape validation specifically.tests/integration/test_workflow.pyruns the real Snakemake DAG against live CMEMS and is@pytest.mark.integration, credential-gated, liketest_cmems_integration.py/test_era5_integration.py— none of the three run on every PR.
Development¶
pixi install
pixi run test-unit # offline unit + input-format tests
pixi run test # same, minus any @pytest.mark.integration test
pixi run dry-run # sanity-check the DAG, no credentials needed
pixi run lint
export CMEMS_USERNAME=... CMEMS_PASSWORD=...
pixi run run-demo # build resources/user/dublin-bay -> results/dublin-bay/ (live CMEMS)
pixi run test-integration-live # + the end-to-end workflow test
snakemake ships from bioconda, not conda-forge alone — pixi.toml's channels must include
both, or snakemake-minimal resolves to nothing.