Reading Data

Reading Data#

Between “I have a CSV of measurements” and “I have a prophys model” sits a step usually retyped per project: parse the file, convert the units, decide what the non-detects mean, turn a wind rose into a distribution. prophys.io does that step, producing the objects the rest of the package already consumes.

The CSV, table and wind-rose readers need nothing beyond NumPy. The raster and NetCDF readers need the geo extra and say so when it is missing.

Units are declared, never guessed. Every reader takes the unit the file is written in and the unit the model works in, and converts between them through prophys.units — so the unit on a model attribute and the unit in the source file are checked against each other exactly once, at the boundary.

Laboratory records#

A results sheet typically contains three kinds of entry: a measurement, a non-detect below an instrument’s limit, and a blank. Substituting a number for either of the last two changes the likelihood the fit maximizes. read_csv() classifies them instead:

site,arsenic,depth
A,0.8,12
B,<0.5,15
C,ND,20
D,1.9,
E,2.4,31

from prophys import io

columns = io.read_csv("lab.csv", unit={"arsenic": "mg/L", "depth": "m"})
columns["arsenic"].summary()
# {'n': 5, 'measured': 3, 'censored': 1, 'missing': 1, 'mean': 1.7, ...}

<0.5 and <=0.5 become left-censored at 0.5, and >30, >=30 become right-censored at 30 (an instrument or scale that saturated); ND with no limit becomes missing, because a censoring term needs a limit to integrate up to; the blank becomes missing; the text column is skipped.

read_observations() goes straight to a fit-ready record:

obs = io.read_observations("lab.csv", "arsenic", unit="ug/L", to_unit="mg/L")
result = prp.finetune(compiled, {"y": obs}, n_steps=3000)

See Observations for what each kind contributes to the likelihood.

Wind and wave roses#

read_wind_rose() reads either shape of directional table — one row per measurement, or one row per sector with a frequency column — and returns directions in radians, the convention every circular distribution in this package uses:

rose = io.read_wind_rose("wind.csv", direction_unit="deg", speed_unit="m/s")
climatology = io.directional_mixture(rose, n_sectors=12)

directional_mixture() divides the circle into equal sectors; each becomes one component of a DirectionalMixture, weighted by its share of the rose, with a Weibull intensity fitted to the speeds recorded in that sector.

That per-sector fit is by the method of moments — shape from the coefficient of variation, scale from the mean. It runs per sector with no optimizer, no starting value and no failure mode, which matters because a rose routinely has a sector with a handful of records in it. The result is a starting climatology; refine it against the same data:

result = prp.fit_distribution(climatology, observations, n_steps=2000)

Rasters and gridded fields#

terrain = io.read_raster("terrain.tif", frame=site, unit="m")

The raster’s affine transform supplies the field’s origin and cell size, so the resulting Field lands in frame at the coordinates the file declares. No-data cells become nan unless nodata_fill is given; any interpolation touching them then yields nan, which is the honest result.

read_netcdf() reads one variable, optionally selecting a single position along named dimensions so a large file need not be materialized in full:

wind = io.read_netcdf(
    "reanalysis.nc", "u10", index={"time": 0}, to_unit="m/s"
)

The variable’s own CF-convention units attribute is used when none is given, and checked against one that is: a file saying m s-1 while the caller says kt is a mistake worth catching at the read.

In-memory tables#

read_table() is the counterpart for rows that came from somewhere other than a file — a database cursor, an API response, a test fixture. It takes the same unit arguments and produces the same Column objects, so downstream code cannot tell the two apart.

Empirical distributions#

empirical = io.empirical_from_column(columns["arsenic"])

A kernel-density distribution over the column’s measured values. Censored and missing entries are excluded: an Empirical describes values that were observed, and a detection limit is not one. When the non-detects carry information worth using, fit a censored likelihood instead.

Reading measurement records out of the formats they arrive in.

This module parses a file, converts its units, interprets non detects and turns a wind rose into a directional distribution. It produces the objects the rest of the package consumes: an Observations, a Field, an Empirical or a directional mixture.

The CSV, table and wind rose readers use only the standard library and NumPy. The NetCDF and GeoTIFF readers need the geo extra and name it when it is missing.

Units are declared, not guessed. A reader takes the unit the file is written in and, optionally, the unit to convert into, and converts through prophys.units, so incompatible units fail at the boundary.

class prophys.io.tabular.Column(name, values, missing, censored, unit='', right_censored=None)[source]#

One parsed column of a table: its values, unit and which entries are missing or censored.

missing, censored and right_censored are boolean masks over values, in the file’s row order. A value under a censoring mask is the limit, not a measurement: the true value lies below it (censored, written "<0.5") or above it (right_censored, written ">30").

Parameters:
name#

Column header.

Type:

str

values#

Parsed values, shape (n,). Missing entries hold 0.

Type:

numpy.ndarray

missing#

Mask of entries that were not measured.

Type:

numpy.ndarray

censored#

Mask of left censored entries (below a detection limit).

Type:

numpy.ndarray

unit#

Unit of values.

Type:

str

right_censored#

Mask of right censored entries (above a limit). All false when not given.

Type:

numpy.ndarray | None

to_observations()[source]#

Return the column as an Observations record, carrying censoring and missingness into the likelihood.

Return type:

Observations

to(unit)[source]#

Return the same column with its values converted into unit.

Parameters:

unit (str)

Return type:

Column

summary()[source]#

Return counts and simple statistics (mean, min, max) over the measured entries only.

Return type:

dict[str, Any]

prophys.io.tabular.read_csv(path, *, columns=None, unit='', to_unit=None, delimiter=None)[source]#

Read numeric columns out of a delimited text file.

Non detects with a limit ("<0.5"), entries above the range (">30") and blanks are recorded as left censoring, right censoring and missingness rather than coerced to a number (see Column). A bare "ND" with no limit counts as missing.

Parameters:

  • path (str | Path) – File to read. The first row is the header.

  • columns (Sequence[str] | None) – Column names to read. By default every column with at least one numeric entry is read.

  • unit (str | Mapping[str, str]) – The unit the file is written in, as one string for every column or a mapping from column name to unit.

  • to_unit (str | Mapping[str, str] | None) – Unit to convert into while reading, as a string or a mapping like unit.

  • delimiter (str | None) – Field separator. By default it is sniffed from the first kilobyte of the file, falling back to a comma.

Returns:

A dict from column name to Column, in file order.

Raises:
Return type:

dict[str, Column]

prophys.io.tabular.read_observations(path, column, *, unit='', to_unit=None, detection_limit=None, delimiter=None)[source]#

Read one column of a CSV file as an Observations record ready to fit.

Parameters:

  • path (str | Path) – File to read.

  • column (str) – Name of the column to read.

  • unit (str) – The unit the file is written in.

  • to_unit (str | None) – Unit to convert into while reading.

  • detection_limit (float | None) – If given, every measured value at or below it is also left censored, for files that record a non detect as the limit itself. It is in unit and converted along with the values. Censoring already written in the file ("<0.5", ">30") is kept either way.

  • delimiter (str | None) – Field separator, sniffed when not given.

Returns:

The column as Observations.

Return type:

Observations

prophys.io.tabular.read_wind_rose(path, *, direction_column='direction', speed_column='speed', frequency_column=None, direction_unit='deg', speed_unit='m/s', delimiter=None)[source]#

Read a directional climatology (wind rose, wave rose) from a CSV file.

Two table shapes are recognized. A record table has one row per measurement, with a direction and a speed. A binned table has one row per sector and a frequency_column. Both produce the same result:

{"directions": radians, "speeds": m/s, "weights": ..., "n": ...}

Rows missing a direction or a speed are dropped. Directions come back in radians, the convention every circular distribution in this package uses (VonMises, WrappedGaussian, DirectionalMixture), and speeds in m/s. Weights sum to one, so a binned rose and a record table are interchangeable downstream.

Raises:

  • KeyError – If a named column is not in the file.

  • ValueError – If the frequency column does not sum to a positive value.

Parameters:

  • path (str | Path)

  • direction_column (str)

  • speed_column (str)

  • frequency_column (str | None)

  • direction_unit (str)

  • speed_unit (str)

  • delimiter (str | None)

Return type:

dict[str, Any]

prophys.io.tabular.directional_mixture(rose, *, n_sectors=12, concentration=None, min_per_sector=8)[source]#

Build a DirectionalMixture climatology from a wind rose read by read_wind_rose().

The circle is divided into n_sectors equal sectors, sector 0 centred on 0 radians. Each becomes one component: its weight is the share of the rose falling in it, and its intensity is a Weibull fitted to the speeds recorded in that sector.

Each sector’s Weibull is fitted by the method of moments (shape from the coefficient of variation, scale from the mean) rather than by maximum likelihood, so it needs no optimizer and cannot fail on a sparse sector. The result is a starting climatology: pass it to prophys.engine.fit_distribution() to refine every sector jointly.

A sector with fewer than min_per_sector records uses the Weibull of all speeds instead of its own. concentration defaults to 1 / (pi / n_sectors)^2, one sector half width as one standard deviation, which roughly tiles the circle.

Raises:

ValueError – If the rose is empty, n_sectors is below 2 or the weights sum to zero.

Parameters:
prophys.io.tabular.empirical_from_column(column, *, bandwidth=None)[source]#

Build a kernel density Empirical distribution over a column’s measured values.

Censored and missing entries are excluded, since a detection limit is not an observed value. Fit a censored likelihood with Observations instead when the non detects carry information worth using. bandwidth defaults to Silverman’s rule.

Raises:

ValueError – If fewer than 2 values were measured.

Parameters:
prophys.io.tabular.read_raster(path, frame, *, band=1, unit='', to_unit=None, nodata_fill=None)[source]#

Read a GeoTIFF (or any raster GDAL can read) into a Field.

Needs the geo extra. The raster’s affine transform supplies the field’s origin and cell size, and the rows are flipped so they run upwards, so the field lands in frame at the coordinates the file declares. The raster’s CRS is not checked against frame.

nodata_fill replaces the raster’s declared nodata value. Left as None, those cells become nan and any interpolation touching them yields nan. Supply a fill only when the model has a real value for “no data here”. Values are converted from unit to to_unit when to_unit is given.

Parameters:
prophys.io.tabular.read_netcdf(path, variable, *, unit='', to_unit=None, index=None)[source]#

Read one variable out of a NetCDF file as a NumPy array.

Needs the geo extra (which supplies netCDF4). index maps dimension names to a single position (a time step, a vertical level), selected before the data is read, so a large file with many dimensions need not be loaded in full.

The variable’s own units attribute is used when unit is not given, and is checked against it when both are present, so a file that says m s-1 while the caller says kg fails at the read. Values are converted into to_unit when it is given.

Raises:
Parameters:
Return type:

ndarray

prophys.io.tabular.read_table(rows, *, unit='', to_unit=None, parse_field=None)[source]#

Parse rows held in memory into columns, the counterpart of read_csv().

For rows that come from somewhere other than a file, such as a database cursor, an API response or a test fixture. Column names come from the first row, and columns with no numeric entry are skipped.

Takes the same unit arguments and produces the same Column objects as read_csv(). A custom parse_field takes the field as a string and returns (value, missing, left_censored) or (value, missing, left_censored, right_censored).

Raises:

ValueError – If there are no rows or no numeric columns.

Parameters:
Return type:

dict[str, Column]