Troubleshooting Guide

AI-generated content

This page was generated by AI and has not been fully reviewed by a human. Content may be inaccurate or incomplete. If you find any issues, please create an issue on the GitHub repository.

This guide covers common issues you might encounter while using CORR-Vars and their solutions.

Concepts API Issues

Problem: A cohort refuses to build

Error messages you might see:

• ConceptsApiConfigurationError: The Concepts API requires a project on every read.

• ConceptsApiConfigurationError: ... CORR_CONCEPTS_API_KEY ...

• ProjectNotFoundError: 'my_project'

• ConceptsLicenseError

Credentials are checked while the cohort is constructed, before any data is loaded, so these surface immediately rather than as a variable that resolves to nothing.

Solutions:

1. Pass a project and a key:

export CORR_CONCEPTS_API_KEY="corr_..."

cohort = Cohort(
    obs_level="icu_stay",
    load_default_vars=False,
    sources={"reprodicu": {"path": "/path/to/reprodICU_files"}},
    project="my_project",          # required by the API on every read
    # api_key="corr_...",          # or CORR_CONCEPTS_API_KEY
)

2. Check which endpoint a source is routed to:

print(cohort.concepts.route("reprodicu"))

Routing is declared in src/corr_vars/concepts/routing.toml and can be overridden per cohort with Cohort(concepts_api_url=...).

3. A 403 means the project’s licence was not accepted — there is no corr_vars-side remedy; a project lead has to accept it.

Problem: Source data cannot be read

Error messages you might see:

• FileNotFoundError when a source reads its data files

• ValueError: No path specified for variable ...

Solutions:

Check the source’s own configuration. For reprodICU that is the path to the dataset files:

cohort = Cohort(
    sources={"reprodicu": {"path": "/path/to/reprodICU_files"}},
    project="my_project",
)

Variable Loading Issues

Problem: Variable not found

Error: ValueError: Variable 'my_variable' not found

Solutions:

1. Check variable name spelling:

# Browse the catalogue in the Concepts Browser web app

# Or list the variable names the Concepts API serves for a source.
# This is a remote read, so the cohort needs its project and key.
configs = cohort.to_search_widget(include_sources=["reprodicu"])

# Names come back fully qualified, e.g. "corr_v1/blood_sodium"
search_term = "sodium"
matching = [v for v in configs["reprodicu"] if search_term in v.lower()]
print(f"Variables containing '{search_term}': {matching}")

2. Check if variable exists for your observation level:

# Some variables are only available for certain observation levels
cohort_icu = Cohort(obs_level="icu_stay", load_default_vars=False)
cohort_hospital = Cohort(obs_level="hospital_stay", load_default_vars=False)

# Try adding the variable to different observation levels
try:
    cohort_icu.add_variable("your_variable")
    print("Variable available for ICU stays")
except:
    print("Variable not available for ICU stays")

Problem: Variable extraction takes too long

Solutions:

1. Use time constraints to limit data:

# Instead of loading all data
cohort.add_variable("blood_sodium")  # Might be slow

# Load only recent data
cohort.add_variable(
    "blood_sodium",
    tmin=("icu_admission", "-24h"),
    tmax=("icu_admission", "+48h")
)

2. Filter your cohort first:

# Filter cohort before adding expensive variables
cohort = Cohort(
    sources={
        "reprodicu": {
            "path": "/path/to/reprodICU_files",
            "include_datasets": ["mimic"],  # Only one dataset
        }
    },
    project="my_project",
)


# Apply inclusion criteria early
cohort.include_list([
    {"variable": "age_on_admission", "operation": ">= 18", "label": "Adults"}
])

# Then add expensive variables
cohort.add_variable("complex_variable")

3. Use load_default_vars=False for large cohorts:

# Don't load all default variables automatically
cohort = Cohort(obs_level="icu_stay", load_default_vars=False)

# Add only the variables you need
needed_vars = ["age_on_admission", "sex", "inhospital_death"]
for var in needed_vars:
    cohort.add_variable(var)

Memory Issues

Problem: Out of memory errors

Error messages: - MemoryError: Unable to allocate array - Kernel died (in Jupyter)

Solutions:

1. Work with smaller cohorts:

# Restrict the cohort to fewer datasets
cohort = Cohort(
    sources={
        "reprodicu": {
            "path": "/path/to/reprodICU_files",
            "include_datasets": ["mimic"],
        }
    },
    project="my_project",
)

2. Process variables in batches:

# Instead of loading all variables at once
all_vars = ["var1", "var2", "var3", "var4", "var5"]

# Load in smaller batches
batch_size = 2
for i in range(0, len(all_vars), batch_size):
    batch = all_vars[i:i+batch_size]
    for var in batch:
        cohort.add_variable(var)

    # Optionally save intermediate results
    cohort.save(f"cohort_batch_{i//batch_size}.corr2")

3. Use Polars instead of Pandas (new API):

# Use the new Polars-based API (more memory efficient)
from corr_vars import Cohort  # New API

# Instead of legacy pandas API
# from corr_vars.legacy_v1 import Cohort  # Legacy API

4. Monitor memory usage:

import psutil
import os

def check_memory():
    process = psutil.Process(os.getpid())
    memory_mb = process.memory_info().rss / 1024 / 1024
    print(f"Memory usage: {memory_mb:.1f} MB")

# Check memory before/after operations
check_memory()
cohort.add_variable("large_variable")
check_memory()

Data Quality Issues

Problem: Unexpected or missing values

Solutions:

1. Inspect data quality:

# Check for missing data
missing_counts = cohort.obs.null_count()
print("Missing data per column:")
print(missing_counts)

# Check data ranges
numeric_cols = cohort.obs.select(pl.col(pl.NUMERIC_DTYPES)).columns
for col in numeric_cols:
    stats = cohort.obs[col].describe()
    print(f"\n{col} statistics:")
    print(stats)

2. Validate time-series data:

# Check dynamic variable data quality
sodium_data = cohort.obsm["blood_sodium"]

print(f"Total sodium measurements: {len(sodium_data)}")
print(f"Patients with sodium data: {sodium_data['icu_stay_id'].n_unique()}")
print(f"Value range: {sodium_data['value'].min()} - {sodium_data['value'].max()}")

# Check for impossible values
impossible = sodium_data.filter(
    (pl.col("value") < 100) | (pl.col("value") > 200)
)
print(f"Impossible sodium values: {len(impossible)}")

3. Apply data cleaning:

# Override a served definition with project-local cleaning bounds
cohort.add_variable_definition(
    "blood_sodium",
    {"cleaning": {"value": {"low": 120, "high": 180}}},  # Physiologically plausible
)
cohort.add_variable("blood_sodium")

Time-Related Issues

Problem: Incorrect time filtering

Solutions:

1. Debug time constraints:

# Check your time anchor columns
print("Available time columns:")
time_cols = [col for col in cohort.obs.columns if 'time' in col.lower() or 'date' in col.lower()]
print(time_cols)

# Verify time anchor values
print(f"ICU admission range: {cohort.obs['icu_admission'].min()} to {cohort.obs['icu_admission'].max()}")

2. Test time constraints step by step:

# Simple time constraint first
cohort.add_variable(
    "blood_sodium",
    save_as="sodium_full_stay",
    tmin="icu_admission",
    tmax="icu_discharge",
)

# Then try relative constraints
cohort.add_variable(
    "blood_sodium",
    save_as="sodium_first_24h",
    tmin="icu_admission",
    tmax=("icu_admission", "+24h"),
)

Problem: Time zones and timestamp issues

Solutions:

# Check timestamp formats
print("Timestamp columns info:")
for col in ["icu_admission", "icu_discharge"]:
    if col in cohort.obs.columns:
        sample_vals = cohort.obs[col].head()
        print(f"{col}: {sample_vals}")

# Ensure consistent time zones (data should be in UTC or local hospital time)
# CORR-Vars handles this automatically, but check if times look reasonable

Custom Variable Issues

Problem: Custom function not working

Error: AttributeError: module has no attribute 'my_function'

Solutions:

1. Check function definition location:

# For an API-routed source the function is part of the published concept and
# is served with it — there is no variables.py in the package to inspect.
# Check the definition in the Concepts Browser, and check that every name it
# uses is declared in the source's exec namespace:
from corr_vars.concepts.compile import get_py_compiler
from corr_vars.sources import get_src_module

py_env = get_src_module("my_source", "py_env")
print(py_env.NAMESPACE_NAMES)  # names a served snippet may use

# A `NameError` from inside a variable function usually means the snippet
# reads something the namespace does not carry. `get_py_compiler(source)`
# returns None for a source that ships no py_env at all.

2. Test custom function separately:

# Test your function with sample data
def my_custom_function(var, cohort):
    # Your function logic here
    print(f"Processing variable: {var.var_name}")
    print(f"Required variables: {var.requires}")
    print(f"Cohort size: {len(cohort.obs)}")

    # Return test data
    import polars as pl
    return pl.DataFrame({
        "icu_stay_id": cohort.obs["icu_stay_id"],
        "test_value": [1.0] * len(cohort.obs)
    })

# Test with a simple variable
from corr_vars.sources.reprodicu import Variable
test_var = Variable(
    var_name="test_custom",
    requires=["age_on_admission"],
    complex=True,
    dynamic=False,
    py=my_custom_function,
    py_ready_polars=True,
)
cohort.add_variable(test_var)

Performance Issues

Problem: Queries are very slow

Solutions:

1. Optimize database queries:

# Restrict the source to fewer datasets to reduce data volume
cohort = Cohort(
    sources={
        "reprodicu": {
            "path": "/path/to/reprodICU_files",
            "include_datasets": ["mimic"],
        }
    },
    project="my_project",
)

2. Load variables strategically:

# Load cheap variables first for filtering
cohort.add_variable("age_on_admission")  # Fast
cohort.add_variable("sex")               # Fast

# Apply filters
cohort.include_list([
    {"variable": "age_on_admission", "operation": ">= 18", "label": "Adults"}
])

# Then load expensive variables on smaller cohort
cohort.add_variable("complex_time_series_variable")  # Slower

3. Use caching:

# Save intermediate results
cohort.save("cohort_after_filtering.corr2")

# Load saved cohort instead of re-extracting
cohort = Cohort.load("cohort_after_filtering.corr2")

File and Save/Load Issues

Problem: Cannot save or load cohort files

Solutions:

1. Check file permissions and paths:

# Check if directory is writable
touch test_file.txt
rm test_file.txt

# Use absolute paths
import os
save_path = os.path.abspath("my_cohort.corr2")
print(f"Saving to: {save_path}")

2. Handle large file sizes:

# For very large cohorts, save without some variables
large_vars = ["variable_with_millions_of_rows"]

# Temporarily remove large variables
saved_data = {}
for var in large_vars:
    if var in cohort.obsm:
        saved_data[var] = cohort._obsm.pop(var)

# Save smaller cohort
cohort.save("cohort_without_large_vars.corr2")

# Restore variables
cohort._obsm.update(saved_data)

3. Migrate from old file formats:

# Load old format and save as new
try:
    old_cohort = Cohort.load("old_file.corr")  # Legacy format
    old_cohort.save("new_file.corr2")          # New format
    print("Successfully migrated file format")
except Exception as e:
    print(f"Migration failed: {e}")

Getting Help

When you encounter issues not covered here:

1. Check the logs:

# Enable verbose logging
import logging
logging.basicConfig(level=logging.DEBUG)

# Or configure CORR-Vars logging
cohort = Cohort(
    logger_args={
        "level": logging.DEBUG,
        "colored_output": True
    }
)

2. Create a minimal reproducible example:

# Simplify your code to the minimal case that reproduces the error
from corr_vars import Cohort

cohort = Cohort(
    obs_level="icu_stay",
    load_default_vars=False,
    sources={"reprodicu": {"path": "/path/to/reprodICU_files"}},
    project="my_project",
)

# Try the operation that's failing
cohort.add_variable("problematic_variable")

3. Gather system information:

# Get debug information
cohort.debug_print()

# Check versions
import corr_vars
import polars as pl
import pandas as pd

print(f"CORR-Vars version: {corr_vars.__version__}")
print(f"Polars version: {pl.__version__}")
print(f"Pandas version: {pd.__version__}")

4. Contact support:

• Create an issue on GitHub: CUB-CORR/corr_vars#issues

• Include your minimal reproducible example

• Include the error message and debug information

• Describe what you expected to happen vs. what actually happened

Remember: Most issues can be resolved by starting with a smaller dataset and gradually increasing complexity!