New perturbation
Can the model predict a gene or treatment it did not see during training?
Generalization ladder
Competition Generalist Track
Generalization is a matrix, not one score.
The evaluation separates familiar-context debugging from tests on unseen perturbations, donors, cell states, laboratories, diseases and combinations.
What is being tested
Each harder task asks the model to cross a new boundary
New donor or disease line
Does the prediction hold in a person or disease background kept entirely out of training?
New cell state or laboratory
Can the model transport across differentiation state, batch or an independent laboratory?
New combination
Can it predict interaction beyond the effects of the two single perturbations?
Current result. Only the first same-assay perturbation holdout has been executed. The harder donor, state, laboratory, disease and combination tasks are defined but have no result yet.
G0–G7
Leakage-safe externalization ladder
| Level | Held-out unit | Current state |
|---|---|---|
| G0 | Random-cell split in one context | No result yetDEBUG_ONLY_NOT_PRIMARY |
| G1 | Same context, unseen perturbation target | Available nowIMPLEMENTED_HEPG2 |
| G2 | Unseen gene family or pathway | Available nowREGISTERED_SPLIT_DATA_NOT_EXECUTED |
| G3 | Unseen donor | Designed, not yet completedREGISTERED_DONOR_RESOLVED_TRUTH_MISSING |
| G4 | Unseen cell type or differentiation state | Designed, not yet completedREGISTERED_MATCHED_TRUTH_MISSING |
| G5 | Unseen dataset or laboratory | Designed, not yet completedREGISTERED_HARMONIZED_TRUTH_MISSING |
| G6 | Unseen disease background | Designed, not yet completedREGISTERED_DMD_PERTURBATION_TRUTH_MISSING |
| G7 | Unseen modality or combination | Designed, not yet completedREGISTERED_OBSERVED_DOUBLE_TRUTH_MISSING |
Permanent controls
Every advanced model must beat the Baseline Zoo
zero_change
Available nowIMPLEMENTED_NATIVEtraining_response_mean
Available nowIMPLEMENTED_NATIVEridge_residual
Available nowIMPLEMENTED_NATIVEnearest_neighbour_transfer
Available nowREGISTERED_NOT_IMPLEMENTEDpathway_module_mean
Available nowREGISTERED_NOT_IMPLEMENTEDsimilarity_weighted_pseudobulk_transfer
Available nowREGISTERED_NOT_IMPLEMENTEDadditive_double_perturbation
Available nowIMPLEMENTED_NULL_PREVIEW_NO_OBSERVED_DOUBLERegistered architecture · not trained
Context-set encoder + perturbation encoder + residual and distribution heads
Context cellsdonor · disease · state · assay
Shared encodercell set and gene modules
Pseudobulk headmean delta · logFC · DEG probability
Distribution headpredicted population · responders · state proportions
AbstentionOOD and uncertainty-aware refusal
No model claim. This architecture is a registered development target. No context-set or generative population model has been trained in this release.
Four leaderboards
Official, generalist, robustness and DMD biology remain separate
DES · PDS · MAE
Designed, not yet completedPARTIAL_AGGREGATE_ADAPTER_DES_LOCKEDDES · PDS · MAE · delta Pearson · logFC Spearman · DEG AUPRC · effect-size Spearman
Designed, not yet completedPARTIAL_AGGREGATE_ADAPTERG1–G7 · calibration · interval coverage · abstention risk · worst-decile performance · per-pathway performance · donor fairness · compute and memory
Designed, not yet completedREGISTERED_DATA_MISSINGfusion · myotube morphology · membrane integrity · calcium · viability · donor replication · reagent replication · disease-specific interaction
Designed, not yet completedREGISTERED_NO_DMD_OUTCOMESError Atlas
Classify how a model fails
common_response_collapsewrong_perturbation_identitycorrect_direction_wrong_magnitudecorrect_mean_wrong_distributiondonor_failurestate_failuretoxicity_missood_overconfidence
Competition task adapter
Wait for official schema
Input, output, cell scale, gene universe, splits, metric weights and submission format remain null until an official task specification is published.
Evaluation dependency
Cell-Eval target pin 0.8.2
The current site runs an internal formula-aligned aggregate adapter, not the official AnnData suite. Version pinning and metric golden tests are required before a competition result is claimed.
Boundary. Registered tasks and model architecture are not benchmark results. Missing donor, state, disease and cell-distribution truth remains null and locked.