Data Sources

CORR-Vars supports multiple data sources for extracting clinical data. Each source provides specialized access to different types of healthcare data.

Available Sources

• reprodICU Data Source
  • Overview
  • Included Datasets
  • Architecture: Axioms and Concepts
  • Configuration
  • Example Usage
  • Future Development
  • Best Practices
  • Class Reference
• Local Data Source (skeleton)
  • The two seams
  • Worked example: a folder of parquet files
  • Configuration
  • Variable definitions
  • Class Reference
• Aggregated Variables
  • Overview
  • Variable Types
  • Declarative form
  • Time Constraints and Filtering
  • Advanced Examples
  • Best Practices
  • Class Reference

Overview

Data sources in CORR-Vars are modular components that handle the extraction and preprocessing of clinical data from different healthcare systems. Each source implements a standardized interface while providing source-specific optimizations and features.

ReprodICU

Parquet-based source for the reprodICU collection of intensive care datasets. It is the ready-to-use example source: point path at the files and it works.

Local data source

The source skeleton. It carries the variable classes and the extraction pipeline but no connection to a store, so a deployment binds it to its own — see Local Data Source (skeleton). Its aggregation and derivation classes need no binding and work on any cohort (Aggregated Variables).

A source is a package under src/corr_vars/sources/, discovered automatically. See Quick Start for what a source has to provide.

Multi-Source Configuration

You can combine multiple data sources in a single cohort:

from corr_vars import Cohort

# Configure multiple sources
cohort = Cohort(
    obs_level="icu_stay",
    sources={
        "reprodicu": {"path": "/path/to/reprodICU_files"},
        # "my_source": {...},
    },
    project="my_project",
)

# Data source tracking
print(cohort.obs["data_source"].value_counts())

Source-Specific Variables

Different sources may provide different variables. The variable loader automatically handles source selection:

# A variable one source publishes and another does not
cohort.add_variable("apache_ii_score")

# Variables are automatically tagged with their source
print("Available sources for this variable:")
print(cohort.obsm["blood_pressure"]["data_source"].unique())

Variable Loader

The variable loader provides the core functionality for loading variables from configured sources:

corr_vars.sources.var_loader.load_variable(var_name, cohort, time_window, include_sources=None, overrides=None, spec=None)

Load var_name from all configured sources and wrap it in a MultiSourceVariable.

Orchestrates config loading, project-override merging, time-window transfer, compatibility checks, and per-source variable instantiation.

Parameters:

• var_name (str) – Name of the variable to load.

• cohort (Cohort) – Cohort that supplies source configs and project overrides.

• time_window (TimeWindow) – Time window for variable extraction.

• include_sources (Iterable[str] | None) – Restrict to these sources. All cohort sources are used when None.

• overrides (dict[str, Any] | None) – Config fields merged onto every source’s vars.json entry, for this load only. Applied after the cohort’s project-local overrides, so these win.

• spec (VariableSpec | None) – Resolved variable reference deciding which taxonomy and version API-routed sources serve. Defaults to the cohort-wide default.

Returns:

Variable ready for extraction.

Return type:

MultiSourceVariable

Raises:

• VariableNotFoundError – If var_name is not found in any source config, or if no source variable could be instantiated.

• AmbiguousConceptError – If spec pins a version and the name resolves to a group of concepts.

• ConceptsApiError – If an API-routed source could not be served.

Base Sources Module