# VBN Manuscript Code

This repository documents the code run for the Visual Behavior Neuropixels
(VBN) manuscript. It is organized around the figure-generation workflow for the
paper: notebooks assemble the figures, utilities provide shared analysis and
data-building code, and cluster scripts generate computationally heavy
intermediate products used by the notebooks.

If you are interested in browsing the Visual Behavior Neuropixels dataset or starting your own analysis, please consult our [`data access tutorials`](https://allensdk.readthedocs.io/en/latest/visual_behavior_neuropixels.html). These notebooks demonstrate how to access and perform basic analyses on the spiking, LFP, and raw electrophysiology data as well as the behavior data/videos associated with this resource.

We have also compiled a [`databook`](https://allenswdb.github.io/physiology/ephys/visual-behavior/VB-Neuropixels.html) to document the tables returned by the AllenSDK in detail.

Note that the notebooks in this repo largely use intermediate data structures derived from the NWB cache (i.e. accessory tables, spike tensors, decoder outputs, Facemap
outputs, and GLM fits). These derived assets are not stored in this repository, but the code to regenerate them is provided or linked.

## Repository Contents

The repository has three main pieces, plus package metadata:

| Path | Contents |
| --- | --- |
| [`vbn_code/Figure notebooks`](vbn_code/Figure%20notebooks) | Notebooks used to produce the main, supplemental, and methods figures in the manuscript. |
| [`vbn_code/utilities`](vbn_code/utilities) | Reusable analysis helpers and scripts that build smaller intermediate tables, tensors, and summaries consumed by the notebooks. |
| [`vbn_code/hpc_code`](vbn_code/hpc_code) | Scripts run on a compute cluster to generate larger or more computationally expensive intermediate products, including decoding outputs, RF arrays, Facemap decoding results, and GLM PSTH predictions. |
| [`pyproject.toml`](pyproject.toml) | Python version, dependencies, and editable-install module list. |

## Environment

The project targets Python 3.8. The package metadata pins this
explicitly with `requires-python = "==3.8.*"` in `pyproject.toml`.

Recommended setup with `conda`:

```bash
conda create -n vbn_manuscript python=3.8
conda activate vbn_manuscript

python -m pip install -e .
```

Optional: register the environment as a Jupyter kernel for running the notebooks
from Jupyter or VS Code.

```bash
python -m pip install ipykernel
python -m ipykernel install --user \
  --name vbn_manuscript \
  --display-name "Python (vbn_manuscript)"
```

The editable install exposes the utility modules in `vbn_code/utilities`, such
as `vbn_utils`, `notebook_utils`, `decoding_utils`, and the table/data builders.

The codebase is standardized on the AllenSDK
`visual-behavior-neuropixels-0.5.0` cache. Builder scripts default to the
manifest name:

```text
visual-behavior-neuropixels_project_manifest_v0.5.0.json
```

## Figure Notebooks

The notebooks in [`vbn_code/Figure notebooks`](vbn_code/Figure%20notebooks)
correspond to the manuscript's figure panels and methods analyses. They are the
main record of the plotting and figure-level analysis code run for the paper.

To run them:

1. Install the environment above.
2. Prepare or obtain the required assets listed below.
3. Open the notebooks with the `vbn_manuscript` kernel.
4. Point notebook path variables to your local cache, supplemental-data,
   derived-data, and external input locations.

## Data Path Conventions

The code refers to three main local data locations:

| Name | Meaning |
| --- | --- |
| `$CACHE_DIR` | Local AllenSDK VBN cache root for `visual-behavior-neuropixels-0.5.0`. |
| `$SUPPLEMENTAL_DIR` | Output directory for regenerated supplemental tables, tensors, and pickles. |
| `$DERIVED_DIR` | Output directory for compute-heavy decoding/model outputs. |

Most builder scripts accept explicit `--cache-dir`, `--output-file`, and input
file arguments.

## Recommended Asset Build Order

The following order reflects the dependency structure among the intermediate
products used by the notebooks:

1. Build `master_stim_table_no_filter.csv`.
2. Build `master_sessions_table.csv`.
3. Build `master_unit_table.csv`.
4. Build active and passive spike tensors.
5. Build optotagging, RF, and responsiveness intermediate tables.
6. Build `master_units_with_responsiveness.csv` from audited unit-table inputs.
7. Build event-aligned, change/prechange, running-matched, modulation, latency,
   and flash-decoding summaries.
8. Build compute-heavy decoding/model outputs where needed.
9. Supply documented external inputs that cannot be regenerated from the VBN cache.

## Intermediate Products

The tables below summarize the intermediate products used by the manuscript
notebooks, grouped by how they are generated or supplied.

### Core Supplemental Assets

These assets are inputs to one or more figure notebooks.

| Asset | Main consumers | Generator or source |
| --- | --- | --- |
| `master_stim_table_no_filter.csv` | Most figure notebooks | [`build_master_stim_table.py`](vbn_code/utilities/build_master_stim_table.py) |
| `master_sessions_table.csv` | Figures 1, 6, S7, S14, Methods | [`build_master_sessions_table.py`](vbn_code/utilities/build_master_sessions_table.py) |
| `master_unit_table.csv` | Unit-table pipeline | [`build_master_unit_table.py`](vbn_code/utilities/build_master_unit_table.py) |
| `master_units_with_responsiveness.csv` | Most figure notebooks | [`build_augmented_master_units_table.py`](vbn_code/utilities/build_augmented_master_units_table.py) |
| `vbnAllUnitSpikeTensor.hdf5` | Figures 2, 4/5, 6, 7, S5, S6, S7 | [`make_unit_tensor.py`](vbn_code/utilities/make_unit_tensor.py) with `--condition active` |
| `vbnAllUnitSpikeTensor_passive.hdf5` | Figures 3, 4/5, S7 | [`make_unit_tensor.py`](vbn_code/utilities/make_unit_tensor.py) with `--condition passive` |
| `units_with_rf_stats.csv` | Figure 1 | [`make_unit_rf_table.py`](vbn_code/utilities/make_unit_rf_table.py) |
| `rfs/arrays/*.npy` | Figure 1 | [`run_rf_arrays.py`](vbn_code/hpc_code/run_rf_arrays.py) |
| `ccf_structure_tree_2017.csv` | Multiple notebooks | Allen CCF reference input. |
| `annotation_25.nrrd` | Figure S3 and unit-layer derivation | Allen CCF reference input, or AllenSDK `ReferenceSpaceCache`. |

### Unit-Table Intermediate Assets

These assets feed `master_units_with_responsiveness.csv`, the unit table used by
the figure notebooks and helper modules.

| Asset | Generator or source | Notes |
| --- | --- | --- |
| `unit_opto_metrics.csv` | [`build_opto_metrics.py`](vbn_code/utilities/build_opto_metrics.py) | Upstream optotagging metrics for audited cell-type labels. |
| `unit_responsiveness_table.csv` | [`build_unit_responsiveness_table.py`](vbn_code/utilities/build_unit_responsiveness_table.py) | Responsiveness columns used by figures/helpers. |
| `unit_rf_table.csv` | [`make_unit_rf_table.py`](vbn_code/utilities/make_unit_rf_table.py) | RF metrics if merged into units. |
| `cluster_labels_122024.csv` | [`vbn_clustering.py`](vbn_code/utilities/vbn_clustering.py) | Cluster labels were generated with the divisive clustering utilities in this file. |

### Notebook Intermediate Assets

These derived products are consumed by figure notebooks and generated by scripts
in [`vbn_code/utilities`](vbn_code/utilities).

| Asset | Notebook consumer | Builder |
| --- | --- | --- |
| `lick_aligned_unit_data.pkl` | Figure 2 | [`build_event_aligned_unit_summaries.py`](vbn_code/utilities/build_event_aligned_unit_summaries.py) |
| `run_start_aligned_unit_data.pkl` | Figure 2 | [`build_event_aligned_unit_summaries.py`](vbn_code/utilities/build_event_aligned_unit_summaries.py) |
| `change_prechange_responses_active_passive_by_unitid.pkl` | Figures 4/5 | [`build_change_prechange_response_summaries.py`](vbn_code/utilities/build_change_prechange_response_summaries.py) |
| `active_passive_baseline_running_matches_nonshared_change_prechange.pkl` | Figures 4/5 | [`build_running_matched_trial_indices.py`](vbn_code/utilities/build_running_matched_trial_indices.py) |
| `change_prechange_responses_active_passive_run_matched_by_unitid.pkl` | Figures 4/5 | [`build_change_prechange_response_summaries.py`](vbn_code/utilities/build_change_prechange_response_summaries.py) with running-matched inputs |
| `change_modulation_across_*.pkl` | Figures 4/5 | [`build_modulation_summary_tables.py`](vbn_code/utilities/build_modulation_summary_tables.py) |
| `state_modulation_across_*.pkl` | Figures 4/5 | [`build_modulation_summary_tables.py`](vbn_code/utilities/build_modulation_summary_tables.py) |
| `novelty_modulation_across_*.pkl` | Figures 4/5 | [`build_modulation_summary_tables.py`](vbn_code/utilities/build_modulation_summary_tables.py) |
| `responsiveness_over_time_dict.pkl` | Figure 6 | [`build_responsiveness_over_time_summary.py`](vbn_code/utilities/build_responsiveness_over_time_summary.py) |
| `latencies_by_region_and_celltype.pkl` | Figure 6 | [`build_latency_summary_by_region_celltype.py`](vbn_code/utilities/build_latency_summary_by_region_celltype.py) |
| `flash_decoding_metrics/*_responseWin_20to100.csv` | Figure 7 | [`build_flash_decoding_metrics.py`](vbn_code/utilities/build_flash_decoding_metrics.py) |
| `flash_decoding_metrics/previous_image_and_change_confidence_response_metrics.csv` | Figure 7 | [`build_flash_decoding_metrics.py`](vbn_code/utilities/build_flash_decoding_metrics.py) |
| `unit_glm_predicted_psths.csv` | Figures S5/S13/S14 | [`aggregate_glm_prediction_psths.py`](vbn_code/utilities/aggregate_glm_prediction_psths.py) |

### Compute-Heavy Derived Outputs

These assets are expected to be generated outside the figure-notebook workflow,
often on a compute cluster.

| Asset | Notebook consumer | Generator |
| --- | --- | --- |
| `VBN_decoding_from_sensory_action_clusters/sessionChangeDecoding_basesub/*.npy` | Figure 3 | [`run_session_decoding.py`](vbn_code/hpc_code/run_session_decoding.py) |
| `VBN_decoding_from_sensory_action_clusters/with_unitsamp_replacement_and_same_splits_for_sessionunits/pooled*/*.npy` | Figure 3 | [`run_pooled_decoding.py`](vbn_code/hpc_code/run_pooled_decoding.py) |
| `image_decoding/*_nonchangeRS.npy` | Figure 6/S13 | [`run_image_decoding.py`](vbn_code/hpc_code/run_image_decoding.py) |
| `image_decoding_sliding_window/*_throughomissionRS_*.npy` | Figure S14_2 | [`run_image_decoding_sliding_window.py`](vbn_code/hpc_code/run_image_decoding_sliding_window.py) |
| `VBN_revision_decoding_dropouts/pooled*/*.npy` | Figures 3, S5/S13/S14 | [`run_decoding_dropouts.py`](vbn_code/hpc_code/run_decoding_dropouts.py) |
| `VBN_revision_decoder_area_comparison_nulls/*.npy` | Figure 3 | [`run_decoder_area_comparison_monte_carlo.py`](vbn_code/hpc_code/run_decoder_area_comparison_monte_carlo.py) |
| `VBN_revision_glm_prediction_psths/*.npz` | Figures S5/S13/S14 | [`run_GLM_prediction_psths.py`](vbn_code/hpc_code/run_GLM_prediction_psths.py), requires external GLM fit files |
| `VBN_video_analysis/facemapDecoding_basesub/*.npy` | Figure 3 | [`run_facemap_decoding.py`](vbn_code/hpc_code/run_facemap_decoding.py), requires external Facemap files |

### Assets That Must Be Supplied Or Documented

Some notebook inputs cannot currently be rebuilt from the AllenSDK VBN cache and
the code in this repository alone.

| Asset | Consumers | Status |
| --- | --- | --- |
| `GLM_dropout_with_unit_info_active_passive.csv` | Figure 2, Figure S3 | Upstream GLM dropout summary generated with [`AllenInstitute/visual_behavior_glm_NP`](https://github.com/AllenInstitute/visual_behavior_glm_NP). |
| `VBN_video_analysis/facemapData/*.hdf5` and validated video/timestamp path table | Figure 3 and Facemap decoding | Upstream behavior video analysis inputs. Raw video data is available at `s3://allen-brain-observatory/visual-behavior-neuropixels/raw-data/` |
| Four-day NWBs and metadata | Figure S13 | Non-release/non-VBN input. |
| Methods behavior summary pickles | Methods | Behavior-model fit summaries generated with [`AllenInstitute/licking_behavior_NP`](https://github.com/AllenInstitute/licking_behavior_NP). |
| `VBN_opto/mouse_opto_fa_rates.npy`, `mouse_opto_hit_rates.npy`, `opto_time.npy` | Figure 3 | Opto behavior analysis inputs. |
| GLM `.pbz2` fit files | GLM PSTH generation | Upstream GLM fit inputs generated with [`AllenInstitute/visual_behavior_glm_NP`](https://github.com/AllenInstitute/visual_behavior_glm_NP). |