Reading Data
Between “I have a CSV of measurements” and “I have a prophys model” sits a
step usually retyped per project: parse the file, convert the units, decide
what the non-detects mean, turn a wind rose into a distribution.
prophys.io does that step, producing the objects the rest of the
package already consumes.
The CSV, table and wind-rose readers need nothing beyond NumPy. The raster
and NetCDF readers need the geo extra and say so when it is missing.
Units are declared, never guessed. Every reader takes the unit the file is
written in and the unit the model works in, and converts between them
through prophys.units — so the unit on a model attribute and the unit
in the source file are checked against each other exactly once, at the
boundary.
Laboratory records
A results sheet typically contains three kinds of entry: a measurement, a
non-detect below an instrument’s limit, and a blank. Substituting a number
for either of the last two changes the likelihood the fit maximizes.
read_csv() classifies them instead:
site,arsenic,depth
A,0.8,12
B,<0.5,15
C,ND,20
D,1.9,
E,2.4,31
from prophys import io
columns = io.read_csv("lab.csv", unit={"arsenic": "mg/L", "depth": "m"})
columns["arsenic"].summary()
# {'n': 5, 'measured': 3, 'censored': 1, 'missing': 1, 'mean': 1.7, ...}
<0.5 becomes left-censored at 0.5; ND with no limit becomes missing,
because a censoring term needs a limit to integrate up to; the blank becomes
missing; the text column is skipped.
read_observations() goes straight to a fit-ready record:
obs = io.read_observations("lab.csv", "arsenic", unit="ug/L", to_unit="mg/L")
result = prp.finetune(compiled, {"y": obs}, n_steps=3000)
See Observations for what each kind contributes to the likelihood.
Wind and wave roses
read_wind_rose() reads either shape of directional table —
one row per measurement, or one row per sector with a frequency column — and
returns directions in radians, the convention every circular
distribution in this package uses:
rose = io.read_wind_rose("wind.csv", direction_unit="deg", speed_unit="m/s")
climatology = io.directional_mixture(rose, n_sectors=12)
directional_mixture() divides the circle into equal
sectors; each becomes one component of a
DirectionalMixture, weighted by its share of
the rose, with a Weibull intensity fitted to
the speeds recorded in that sector.
That per-sector fit is by the method of moments — shape from the coefficient of variation, scale from the mean. It runs per sector with no optimizer, no starting value and no failure mode, which matters because a rose routinely has a sector with a handful of records in it. The result is a starting climatology; refine it against the same data:
result = prp.fit_distribution(climatology, observations, n_steps=2000)
Rasters and gridded fields
terrain = io.read_raster("terrain.tif", frame=site, unit="m")
The raster’s affine transform supplies the field’s origin and cell size, so
the resulting Field lands in frame at the
coordinates the file declares. No-data cells become nan unless
nodata_fill is given; any interpolation touching them then yields
nan, which is the honest result.
read_netcdf() reads one variable, optionally selecting a
single position along named dimensions so a large file need not be
materialized in full:
wind = io.read_netcdf(
"reanalysis.nc", "u10", index={"time": 0}, to_unit="m/s"
)
The variable’s own CF-convention units attribute is used when none is
given, and checked against one that is: a file saying m s-1 while the
caller says kt is a mistake worth catching at the read.
In-memory tables
read_table() is the counterpart for rows that came from
somewhere other than a file — a database cursor, an API response, a test
fixture. It takes the same unit arguments and produces the same
Column objects, so downstream code cannot tell the two
apart.
Empirical distributions
empirical = io.empirical_from_column(columns["arsenic"])
A kernel-density distribution over the column’s measured values.
Censored and missing entries are excluded: an Empirical describes values
that were observed, and a detection limit is not one. When the non-detects
carry information worth using, fit a censored likelihood instead.
Reading measurement records out of the formats they arrive in.
Between “I have a CSV of wind measurements” and “I have a prophys model”
sits a step that is usually retyped per project: parse the file, convert the
units, decide what the non-detects mean, turn a wind rose into a directional
distribution. This module does that step, producing the objects the rest of
the package already consumes — an Observations,
a Field, an
Empirical or a directional mixture.
Everything here is deliberately built on the standard library and NumPy only. A CSV reader that needs pandas installed is not usable in the environments this package targets; where a heavier format genuinely requires a dependency (NetCDF, GeoTIFF), the reader says so and names the extra.
Units are declared, not guessed. Every reader takes the unit the file is
written in and the unit the model works in, and converts between them
through prophys.units — so the unit annotation on a model attribute
and the unit in the source file are checked against each other exactly once,
here, at the boundary.
- class prophys.io.tabular.Column(name, values, missing, censored, unit='')[source]
One parsed column of a table: its values and what they mean.
missing and censored are boolean masks over values, in the file’s row order. A value under a censoring mask is the limit, not a measurement.
- prophys.io.tabular.read_csv(path, *, columns=None, unit='', to_unit=None, delimiter=None)[source]
Read numeric columns out of a delimited text file.
Parameters
- columns:
Column names to read; the default is every column whose values parse as numbers.
- unit:
The unit the file is written in — one string for every column, or a per-column mapping.
- to_unit:
Convert into this unit while reading. Incompatible units raise
UnitErrorhere, at the boundary, rather than producing a model that is quietly wrong by a factor of a thousand.- delimiter:
Defaults to sniffing the file’s first kilobyte, falling back to a comma.
Non-detects (
"<0.5","ND") and blanks are recognized and recorded as censoring and missingness rather than being coerced to a number — seeColumn.
- prophys.io.tabular.read_observations(path, column, *, unit='', to_unit=None, detection_limit=None, delimiter=None)[source]
One column of a CSV as a ready-to-fit Observations record.
detection_limit additionally censors every value at or below it, for files that record a non-detect as the limit itself rather than marking it. Censoring already encoded in the file (
"<0.5") is honoured either way.
- prophys.io.tabular.read_wind_rose(path, *, direction_column='direction', speed_column='speed', frequency_column=None, direction_unit='deg', speed_unit='m/s', delimiter=None)[source]
Read a directional climatology (wind rose, wave rose) from a table.
Two table shapes are recognized. A record table has one row per measurement, with a direction and a speed; a binned table has one row per sector with a frequency column. Both produce the same result:
{"directions": radians, "speeds": ..., "weights": ..., "n": ...}Directions come back in radians, the convention every circular distribution in this package uses (VonMises, WrappedGaussian, DirectionalMixture). Weights sum to one, so a binned rose and a record table are interchangeable downstream.
- prophys.io.tabular.directional_mixture(rose, *, n_sectors=12, concentration=None, min_per_sector=8)[source]
Build a DirectionalMixture climatology from a wind rose read by
read_wind_rose().The circle is divided into n_sectors equal sectors. Each becomes one component: its occurrence weight is the share of the rose falling in it, and its conditional intensity is a Weibull fitted to the speeds recorded in that sector.
The per-sector Weibull is fitted by the method of moments — shape from the coefficient of variation, scale from the mean — rather than by maximum likelihood. It runs per sector with no optimizer, no starting value and no failure mode, which matters because a rose routinely has a sector with a handful of records in it. The result is a starting climatology, not a final one: pass it to
prophys.engine.fit_distribution()to refine every sector jointly against the same data.A sector with fewer than min_per_sector records borrows the overall speed distribution rather than fitting its own, since a two-point coefficient of variation is noise. concentration defaults to a spread that roughly tiles the circle at the chosen n_sectors.
- prophys.io.tabular.empirical_from_column(column, *, bandwidth=None)[source]
A kernel-density Empirical distribution over a column’s measured values.
Censored and missing entries are excluded: a Empirical is a description of values that were observed, and a detection limit is not one of them. Fit a censored likelihood with
Observationsinstead when the non-detects carry information worth using.
- prophys.io.tabular.read_raster(path, frame, *, band=1, unit='', to_unit=None, nodata_fill=None)[source]
Read a GeoTIFF (or any GDAL-readable raster) into a Field.
Needs the
geoextra. The raster’s affine transform supplies the field’s origin and cell size, so a field read this way lands in frame at the coordinates the file declares — provided frame’s units match the raster’s CRS units, which is checked.nodata_fill replaces the raster’s declared no-data value. Left as None, no-data cells become nan and any interpolation touching them yields nan, which is the honest result; supply a fill only when the model genuinely has a value for “no data here”.
- prophys.io.tabular.read_netcdf(path, variable, *, unit='', to_unit=None, index=None)[source]
Read one variable out of a NetCDF file as a NumPy array.
Needs the
geoextra (which supplies netCDF4). index selects a single position along named dimensions — a time step, a vertical level — before the array is returned, so a large multi-dimensional file does not have to be materialized in full.The variable’s own
unitsattribute is used when unit is not given, and is checked against it when both are present: a file that saysm s-1while the caller saysktis a mistake worth catching at the read.
- prophys.io.tabular.read_table(rows, *, unit='', to_unit=None, parse_field=None)[source]
The in-memory counterpart of
read_csv(), for rows that came from somewhere other than a file — a database cursor, an API response, a test fixture.Takes the same unit arguments and produces the same
Columnobjects, so downstream code cannot tell the two apart.