Observations

Observations#

calibrate and finetune accept a plain array of observations, which asserts that every value was measured exactly. Real measurement records routinely are not:

  • a concentration below an instrument’s detection limit is known only to be somewhere below it;

  • a specimen that survived the whole test is known only to have failed after the test ended;

  • a reading logged to the nearest bin is known only to lie within it;

  • a failed sensor produced no information at all.

Substituting a number for any of those — the limit, a half-limit, a bin midpoint, an interpolated fill — changes the likelihood the fit maximizes, and so changes the fitted parameters. Observations carries what the record actually says.

Kind

Contributes

Constructor

exact

\(\log f(x)\)

exact()

left-censored

\(\log F(\text{limit})\)

detection_limit()

right-censored

\(\log(1 - F(\text{limit}))\)

right_censored()

interval

\(\log(F(\text{upper}) - F(\text{lower}))\)

interval()

missing

\(0\)

from_arrays(..., missing=mask)

Pass an Observations anywhere an observation array was accepted; the engine dispatches on the type.

Non-detects#

import prophys as prp

obs = prp.Observations.detection_limit(concentrations, limit=0.5)
result = prp.finetune(compiled, {"c": obs}, n_steps=3000)

Values at or below the limit become left-censored at it; the rest stay exact. Compared with substituting the limit for every non-detect, this recovers the generating parameters rather than pulling the mean up and the scale down.

Survival and reliability#

obs = prp.Observations.right_censored(recorded, censored=still_running)

Specimens still unfailed when the test ended are known only to exceed their recorded time. Ignoring the censoring treats every truncated value as a failure, which understates the lifetime and overstates the hazard rate.

Binned readings#

obs = prp.Observations.interval(lower_edges, lower_edges + width)

Every observation known only to lie between two bounds — readings rounded or binned at recording time.

Missing entries#

obs = prp.Observations.from_arrays(values, missing=np.isnan(values))

Marked records contribute nothing, which gives the same fit as omitting them while keeping the record’s length and row order intact. Values at those positions may be nan in the caller’s array; they are replaced by a finite placeholder, so a nan cannot reach a gradient through an unselected branch.

Downstream effects#

An Observations record changes more than the fit:

  • information_criteria() counts only informative observations, so missing records do not inflate the sample-size penalty;

  • parameter_uncertainty() reports the same count;

  • calibration() and crps() use only the exactly-observed values, since a censored record carries no PIT value;

  • RunManifest records the counts by kind, so an exported result says how much of its data was censored.

read_observations() builds one straight from a file — see Reading Data.

class prophys.domain.Observations(value, lower, upper, kind)[source]#

A set of observations of one attribute, each with its own kind.

Build one with the classmethod matching how the data were recorded (exact(), detection_limit(), right_censored(), interval(), from_arrays()) rather than through __init__, which takes the resolved arrays.

Parameters:

  • value (Any) – Observed values, one dimensional. Ignored for kinds other than exact, but must still be finite: a nan here propagates through jnp.where into the gradient even on branches that discard it, so missing and censored entries carry a placeholder instead.

  • lower (Any) – Lower bounds, read by the right censored and interval kinds.

  • upper (Any) – Upper bounds, read by the left censored and interval kinds.

  • kind (Any) – Code per observation, from the module constants (EXACT, LEFT_CENSORED, RIGHT_CENSORED, INTERVAL_CENSORED, MISSING).

Raises:

ValueError – If the arrays differ in shape, are not one dimensional, or value holds a non finite entry.

classmethod exact(values)[source]#

Every value measured exactly, the same likelihood a plain array gives, in Observations form.

Parameters:

values (Any)

Return type:

Observations

classmethod detection_limit(values, limit)[source]#

Values at or below limit are left censored at it, the rest are exact.

This is the standard record of nondetects from an analytical laboratory: the instrument reports “< limit”, and all that is known is that the true value lies somewhere below.

Parameters:
Return type:

Observations

classmethod right_censored(values, censored)[source]#

Entries where censored is true are known only to exceed their recorded value, like a specimen still intact when the test ended.

Parameters:
Return type:

Observations

classmethod interval(lower, upper)[source]#

Every observation known only to lie between lower and upper, like readings rounded or binned at recording time.

Raises:

ValueError – If the bounds differ in shape or any upper is below its lower.

Parameters:
Return type:

Observations

classmethod from_arrays(values, *, lower=None, upper=None, kind=None, missing=None)[source]#

The general constructor, with a kind and bounds per observation.

kind defaults to all exact. missing is a boolean mask whose entries are recoded as MISSING regardless of kind and contribute nothing. Every value the likelihood does not read, and any non finite value, is replaced by 0.0, and nan bounds become 0.0, so a record stored with nan at its gaps can be passed in directly. Omitted bounds default to zeros.

Parameters:
Return type:

Observations

log_prob(compiled, attribute_name, raw_params)[source]#

Log likelihood contribution of each observation under compiled.

Every branch is evaluated for every observation and selected with jnp.where, rather than gathered per kind: the selection has to survive jax.jit and jax.grad, where a Python branch on the data branch would either retrace per record or bake one kind’s rule into the compiled function.

Parameters:

attribute_name (str)

Return type:

Any

counts()[source]#

Number of observations of each kind present, for a fit’s provenance record in the export manifest and for the repr.

Return type:

dict[str, int]

property n_informative: int#

Observations that contribute a likelihood term, that is every one not marked missing. This is the count an information criterion’s sample size should use.

prophys.domain.as_observations(obs)[source]#

Coerce a plain array into exact Observations, returning an Observations unchanged.

Lets code that needs the observation model (an information criterion’s sample size, a provenance record) treat both input forms uniformly.

Parameters:

obs (Any)

Return type:

Observations