# scGPT 0.2.5 frozen-split benchmark

Epochs: 15; seeds: 5; device: cuda:0.

Only checkpoint-supported held-out targets are scored. All comparator models are restricted to identical seed-gene pairs.

## Same-coverage summary

| model | seed | n_test_genes | mean_model_rmse | mean_zero_rmse | mean_train_mean_rmse | mean_model_cosine | mean_train_mean_cosine | mean_perturbation_specific_residual_cosine | relative_rmse_improvement_vs_zero | relative_rmse_improvement_vs_train_mean | raw_cosine_difference_vs_train_mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ridge_v1_same_scgpt_coverage | 20260712 | 63 | 0.10579 | 0.112462 | 0.107973 | 0.331228 | 0.337819 | 0.223294 | 0.0593249 | 0.0202124 | -0.00659051 |
| ridge_v1_same_scgpt_coverage | 20260713 | 67 | 0.104002 | 0.104369 | 0.104711 | 0.281018 | 0.272533 | 0.219553 | 0.00351381 | 0.00677248 | 0.00848492 |
| ridge_v1_same_scgpt_coverage | 20260714 | 55 | 0.107024 | 0.113219 | 0.10938 | 0.30281 | 0.330519 | 0.240318 | 0.0547164 | 0.021537 | -0.0277091 |
| ridge_v1_same_scgpt_coverage | 20260715 | 62 | 0.109771 | 0.112181 | 0.110173 | 0.258406 | 0.300443 | 0.190628 | 0.0214839 | 0.0036474 | -0.0420368 |
| ridge_v1_same_scgpt_coverage | 20260716 | 46 | 0.105583 | 0.113584 | 0.104465 | 0.362871 | 0.405483 | 0.197707 | 0.0704405 | -0.0107018 | -0.0426118 |
| scgpt_0.2.5_whole_human | 20260712 | 63 | 0.114358 | 0.112462 | 0.106151 | 0.287259 | 0.350726 | 0.143793 | -0.0168553 | -0.0773063 | -0.063467 |
| scgpt_0.2.5_whole_human | 20260713 | 67 | 0.109404 | 0.104369 | 0.103102 | 0.266355 | 0.280397 | 0.153794 | -0.0482485 | -0.0611223 | -0.0140419 |
| scgpt_0.2.5_whole_human | 20260714 | 55 | 0.114255 | 0.113219 | 0.107974 | 0.275406 | 0.335141 | 0.156658 | -0.00915271 | -0.0581741 | -0.0597346 |
| scgpt_0.2.5_whole_human | 20260715 | 62 | 0.115477 | 0.112181 | 0.108349 | 0.274921 | 0.305824 | 0.146773 | -0.0293796 | -0.0657803 | -0.030903 |
| scgpt_0.2.5_whole_human | 20260716 | 46 | 0.123606 | 0.113584 | 0.1035 | 0.271219 | 0.414413 | 0.0985081 | -0.0882336 | -0.194258 | -0.143193 |

## Direct scGPT versus ridge_v1 comparison

| seed | comparison | n_genes | mean_difference | bootstrap_ci95_low | bootstrap_ci95_high | direction_if_scgpt_better | paired_randomization_p_two_sided | holm_p_model_comparison_family |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 20260712 | scgpt_minus_ridge_rmse | 63 | 0.00856737 | 0.00427486 | 0.0129413 | less | 0.00019999 | 0.00279986 |
| 20260712 | scgpt_minus_ridge_raw_cosine | 63 | -0.0439691 | -0.0809437 | -0.00812312 | greater | 0.019799 | 0.217789 |
| 20260712 | scgpt_minus_ridge_residual_cosine | 63 | -0.0795008 | -0.170998 | 0.0091771 | greater | 0.0934953 | 0.654467 |
| 20260713 | scgpt_minus_ridge_rmse | 67 | 0.00540236 | -0.000191225 | 0.0110637 | less | 0.0650967 | 0.520774 |
| 20260713 | scgpt_minus_ridge_raw_cosine | 67 | -0.0146633 | -0.0515193 | 0.0234766 | greater | 0.443278 | 1 |
| 20260713 | scgpt_minus_ridge_residual_cosine | 67 | -0.0657595 | -0.158829 | 0.0262974 | greater | 0.183991 | 0.919954 |
| 20260714 | scgpt_minus_ridge_rmse | 55 | 0.00723121 | 0.00173184 | 0.0125848 | less | 0.0117994 | 0.141593 |
| 20260714 | scgpt_minus_ridge_raw_cosine | 55 | -0.0274036 | -0.0755467 | 0.0181333 | greater | 0.277786 | 1 |
| 20260714 | scgpt_minus_ridge_residual_cosine | 55 | -0.0836597 | -0.183717 | 0.0219696 | greater | 0.127494 | 0.764962 |
| 20260715 | scgpt_minus_ridge_rmse | 62 | 0.00570591 | 0.000762637 | 0.0107005 | less | 0.0316984 | 0.316984 |
| 20260715 | scgpt_minus_ridge_raw_cosine | 62 | 0.016515 | -0.0211741 | 0.0542895 | greater | 0.40218 | 1 |
| 20260715 | scgpt_minus_ridge_residual_cosine | 62 | -0.0438545 | -0.151119 | 0.0585888 | greater | 0.40338 | 1 |
| 20260716 | scgpt_minus_ridge_rmse | 46 | 0.0180228 | 0.0126161 | 0.0231001 | less | 9.9995e-05 | 0.00149993 |
| 20260716 | scgpt_minus_ridge_raw_cosine | 46 | -0.0916519 | -0.137962 | -0.0427878 | greater | 0.000899955 | 0.0116994 |
| 20260716 | scgpt_minus_ridge_residual_cosine | 46 | -0.0991986 | -0.189378 | -0.00533518 | greater | 0.0446978 | 0.40228 |