Overview of the VEDB preprocessing, SimCLR pretraining, downstream linear probes, and voxelwise encoding workflow.
## NSD Data and Evaluation Voxelwise encoding models were fit using fMRI responses from the [**Natural Scenes Dataset (NSD)**](https://www.naturalscenesdataset.org/), which contains 7T whole-brain fMRI measurements collected while participants viewed large sets of natural-scene images. The present analysis used data from **8 NSD participants**. For each participant, the **1,000 images shared across all NSD participants** were reserved as the final held-out evaluation set. The remaining approximately **9,000 participant-specific images** were used for model fitting and regularization selection. Within this fitting set, a nested holdout partition was used to select the ridge penalty independently for each voxel. The same intact `224 × 224` NSD images were presented to each of the seven pretrained ResNet-18 models. The Baseline, Fovea-Gaze, Periph, and Periph-NF transformations were applied only during VEDB SimCLR pretraining and were **not reapplied to NSD stimuli** during feature extraction. The same NSD feature-extraction procedure was used for the STL-10, ImageNet-100, and ImageNet-1K reference models. Encoding performance was evaluated on the held-out 1,000-image set using voxelwise coefficient of determination (`R²`) and Pearson correlation (`corr`). These held-out metrics are stored directly in each released encoding-model fit. ## Encoding-Model Procedure The encoding models were fit separately for each **NSD subject (S1–S8)** and each of the seven pretrained visual models. NSD stimuli were presented to the pretrained ResNet-18 models as intact `224 × 224` images. The Baseline, Fovea-Gaze, Periph, and Periph-NF transformations were used during SimCLR pretraining and were **not reapplied to NSD images** during encoding-model evaluation. Before feature extraction, NSD images were rescaled to `[0, 1]` and normalized using ImageNet channel statistics: ```text mean = [0.485, 0.456, 0.406] std = [0.229, 0.224, 0.225] ``` Features were extracted from six points in the ResNet-18 encoder: ```text conv1 layer1.1 layer2.1 layer3.1 layer4.1 avgpool ``` Convolutional feature maps were reduced with adaptive average pooling and then flattened. PCA was applied separately to each layer, retaining the top **200 principal components**. The six 200-component representations were then concatenated before voxelwise model fitting. PCA was fit separately for each subject, model condition, and feature layer. In the analysis used for the study, PCA was applied to the full subject-specific feature matrix before the encoding-model train and holdout partitions were applied. ### Feature Normalization and Ridge Fitting For each subject, the feature matrix was divided into a training partition, a nested holdout partition for ridge-penalty selection, and a final held-out evaluation partition containing the **1,000 NSD images shared across participants**. Feature-wise means and standard deviations were estimated from the combined training and nested-holdout partitions. The same parameters were then used to z-score all three partitions. The final held-out evaluation set was excluded when estimating these normalization parameters. A column of ones was appended to the feature matrix to provide an intercept term. Voxel responses were modeled with **L2-regularized linear regression (ridge regression)**. Twenty candidate ridge penalties were evaluated. The penalty that minimized prediction error on the nested holdout partition was selected independently for each voxel. Performance on the final held-out set was quantified with voxelwise coefficient of determination (`R²`) and Pearson correlation (`corr`). ## Saved Fit Contents Each `.npy` file contains a Python dictionary with the fitted encoding model and associated metadata: - `subject` — NSD subject number - `model` — visual model identifier - `features_file_list` — feature files used for model fitting - `lambdas` — candidate ridge penalties - `weights` — fitted voxelwise ridge-regression weights - `best_lambda_inds` — selected ridge-penalty index for each voxel - `r2` — held-out voxelwise coefficient of determination - `corr` — held-out voxelwise correlation - `voxel_mask` — subject voxel mask - `voxel_index` — voxel indices - `voxel_nc` — voxel noise-ceiling estimates - `brain_nii_shape` — original volumetric brain-image shape Because the intercept was added as a final column in the feature matrix, the **final row of `weights` contains the fitted intercept term**. ## Internal Model Labels The released `.npy` files retain the model identifiers used by the original NSD feature-extraction and encoding-model analysis pipeline. These identifiers are analysis-time labels and should not be interpreted as model names stored within the underlying SimCLR checkpoints. | Public model name | Encoding-analysis identifier | | --- | --- | | Baseline | `resnet18-Baseline` | | Fovea-Gaze | `resnet18-FoveaGaze` | | Periph | `resnet18-PeriphNonTTM` | | Periph-NF | `resnet18-PeriphTTM` | | STL-10 | `resnet18-pretrained-simclr` | | ImageNet-100 | `resnet18-simclr-imgnet100` | | ImageNet-1K | `resnet18-simclr-imgnet1k` | The ImageNet-100 and ImageNet-1K PyTorch Lightning checkpoints themselves store model parameters under `backbone.*` and `projection_head.*` namespaces and do not contain the dataset-specific analysis identifiers shown above. The STL-10 checkpoint obtained from Spijkervet/SimCLR uses its original `encoder.*` and `projector.*` parameter naming, which was remapped during feature extraction. The internal identifier `resnet18-pretrained-simclr` corresponds to the STL-10 SimCLR ResNet-18 checkpoint obtained from the external [Spijkervet/SimCLR](https://github.com/Spijkervet/SimCLR) release. These analysis identifiers are preserved in the `model` field of the released encoding-model fits and, where applicable, in fields such as `model1`, `model2`, and variance-partitioning dictionary keys. They reflect the naming conventions used during model development and do not indicate additional model conditions. The `features_file_list`, `features_file_list1`, and `features_file_list2` fields also preserve the original analysis paths for provenance. These paths refer to the computing environment used for the original study and are not expected to resolve outside that environment. ## Variance-Partitioning Fits The repository also contains the subject-specific variance-partitioning fits reported in the associated paper: - **Fovea-Gaze vs. Periph** — Figure 4C - **Periph vs. Periph-NF** — Figure 4D For each comparison and NSD participant, three voxelwise encoding models were fit using (1) features from model A alone, (2) features from model B alone, and (3) the concatenated feature spaces from both models. Unique variance was computed from held-out voxelwise `R²` using the equations reported in the paper: $$ R^2_{\mathrm{unique},A} = R^2_{\mathrm{combined}} - R^2_{B\text{-only}} $$ $$ R^2_{\mathrm{unique},B} = R^2_{\mathrm{combined}} - R^2_{A\text{-only}} $$ Thus, the unique variance attributed to one model is the variance explained by the combined representation minus the variance explained by the other model alone. Variance-partitioning files are stored under: ```text variance-partitioning/ ├── fovea-gaze-vs-periph/ │ ├── NSD_S1_varpart_Fovea-Gaze_vs_Periph.npy │ ├── ... │ └── NSD_S8_varpart_Fovea-Gaze_vs_Periph.npy └── periph-vs-periph-nf/ ├── NSD_S1_varpart_Periph_vs_Periph-NF.npy ├── ... └── NSD_S8_varpart_Periph_vs_Periph-NF.npy ``` Each variance-partitioning file contains a Python dictionary with: - `subject` - `model1` - `model2` - `features_file_list1` - `features_file_list2` - `lambdas` - `voxel_mask` - `voxel_index` - `voxel_nc` - `brain_nii_shape` - `weights_varpart` - `r2_varpart` - `corr_varpart` - `best_lambda_inds_varpart` The `weights_varpart`, `r2_varpart`, `corr_varpart`, and `best_lambda_inds_varpart` fields are dictionaries with entries for model A alone, model B alone, and the combined feature space. Their exact keys retain the original internal model identifiers listed above. Because the six retained layers contribute 200 PCA components each, a single-model variance-partition fit contains **1,200 model features plus one intercept**, giving `weights_varpart` matrices with 1,201 rows for the model-only fits. The combined fit contains 2,400 model features plus one intercept, giving 2,401 rows. Example: ```python import numpy as np fit = np.load( "variance-partitioning/fovea-gaze-vs-periph/" "NSD_S1_varpart_Fovea-Gaze_vs_Periph.npy", allow_pickle=True ).item() r2 = fit["r2_varpart"] model_a = fit["model1"] model_b = fit["model2"] r2_a_only = r2[f"{model_a}-only"] r2_b_only = r2[f"{model_b}-only"] r2_combined = r2["combined"] r2_unique_a = r2_combined - r2_b_only r2_unique_b = r2_combined - r2_a_only ``` ## Loading a Fit ```python import numpy as np fit = np.load( "baseline/NSD_S1_resnet18-Baseline_concat.npy", allow_pickle=True ).item() weights = fit["weights"] r2 = fit["r2"] corr = fit["corr"] lambdas = fit["lambdas"] best_lambda_inds = fit["best_lambda_inds"] voxel_index = fit["voxel_index"] voxel_nc = fit["voxel_nc"] ``` ## Scope of the Released Artifacts The repository contains **56 subject-specific encoding-model fits** across seven pretrained visual models: **32 fits from the four VEDB-pretrained models** (4 models × 8 NSD participants) and **24 fits from the three non-egocentric reference models** (3 models × 8 NSD participants). The repository additionally contains **16 variance-partitioning fits** corresponding to the two variance-partition analyses reported in the paper (2 model comparisons × 8 participants), for a total of **72 fitted `.npy` artifacts**. The released files contain fitted voxelwise weights, held-out prediction metrics, ridge-penalty information, and voxel metadata used in the study. The variance-partitioning artifacts additionally contain the model-A-only, model-B-only, and combined fits used to compute unique variance. Reproducing voxel predictions for new images also requires the corresponding pretrained ResNet-18 checkpoint and the feature-extraction, spatial-pooling, PCA, concatenation, and normalization procedures used during fitting. These steps will be documented in the accompanying public code release. ## Related Models This repository is associated with the **[Eccentricity-Constrained SimCLR Models (VEDB)](https://hf.co/collections/DM-Diaz/eccentricity-constrained-simclr-models-vedb)** collection. - [VEDB SimCLR ResNet-18 — Baseline](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Baseline) - [VEDB SimCLR ResNet-18 — Fovea-Gaze](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Fovea-Gaze) - [VEDB SimCLR ResNet-18 — Periph](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Periph) - [VEDB SimCLR ResNet-18 — Periph-NF](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Periph-NF) - [VEDB NSD ResNet-18 — Encoding Models](https://huggingface.co/DM-Diaz/VEDB-NSD-ResNet18-Encoding-Models) - [SimCLR ResNet-18 — ImageNet-1K](https://huggingface.co/DM-Diaz/SimCLR-ResNet18-ImageNet1K) - [SimCLR ResNet-18 — ImageNet-100](https://huggingface.co/DM-Diaz/SimCLR-ResNet18-ImageNet100) - [SimCLR ResNet-18 — STL-10](https://github.com/Spijkervet/SimCLR) *(external pretrained reference model; checkpoint provided by Spijkervet/SimCLR and not redistributed by this project)* ## Computational Resources The computational experiments and analyses for this study were conducted primarily using Carnegie Mellon University Neuroscience Institute's [MiND computing cluster](https://ni.cmu.edu/computing/knowledge-base/mind-cluster-nodes/). ## Citation If you use these encoding-model fits in academic work, please cite the associated study: ```bibtex @inproceedings{diaz2026eccentricity, author = {Diaz, Dylan M. and Henderson, Margaret M.}, title = {Eccentricity-Constrained CNN Training Reveals Adaptive Information Coding Around the Visual Field}, booktitle = {Proceedings of the 9th Conference on Cognitive Computational Neuroscience}, address = {New York, NY, USA}, year = {2026}, doi = {10.32470/0416gfsq} } ``` **Proceedings:** Conference on Cognitive Computational Neuroscience 2026 **Preprint:** `arXiv:2607.19316` ### NSD Citation Researchers using the Natural Scenes Dataset should cite: **Allen, E. J., et al. (2022). _A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence._ Nature Neuroscience, 25, 116–126. https://doi.org/10.1038/s41593-021-00962-x** See the [Natural Scenes Dataset paper](https://www.nature.com/articles/s41593-021-00962-x) for the complete author list and dataset description. ### VEDB Citation Researchers using the underlying Visual Experience Dataset should also cite: **Greene, M. R., et al. (2024). _The Visual Experience Dataset: Over 200 recorded hours of integrated eye movement, odometry, and egocentric video._ Journal of Vision, 24(11), 6.** See the [VEDB dataset paper](https://jov.arvojournals.org/article.aspx?articleid=2802101) for the complete author list and citation information. ## License The encoding-model fits in this repository are released under the **Apache License 2.0**. The VEDB dataset and other third-party resources used in the associated study remain subject to their respective licenses, access requirements, and terms of use. This repository does not redistribute the full VEDB dataset; a small number of example frames are included to illustrate the published visual-field manipulations.