## Overview This repository is organized as a reproducible analysis pipeline: 1. Raw data are stored as NWB sessions in `data_electrophysiology/` and `data_optogenetics/`. 2. Helper metadata such as area mappings and lick-time exports live in `data_helpers/`. 3. Notebooks in `processed_data_code/` generate the intermediate `.npz` and `.pkl` files written to `processed_data/`. 4. Notebooks in `Main_figures_codes/` and `Supplementary_figures_codes/` load those processed outputs and export the final PDFs in the corresponding figure folders. ## Pipeline Schema ```text data_electrophysiology/ data_optogenetics/ \ / \ / +---- data_helpers/ + | v processed_data_code/ | v processed_data/ / \ v v Main_figures_codes/ Supplementary_figures_codes/ | | v v Main_figures_pdf/ Supplementary_figures_pdf/ ``` ## Repository Structure The repository is easier to read if you think of it as five layers: 1. raw data 2. shared Python methods 3. dataset-building notebooks 4. saved intermediate outputs 5. final figure notebooks and exported PDFs Top-level folders and files: - `data_electrophysiology/`: raw electrophysiology NWB sessions used by the main neural-analysis workflow. - `data_optogenetics/`: raw optogenetics NWB sessions used by the optogenetic analyses and brainmap figures. - `data_helpers/`: small helper files such as area mappings, top coordinates, and auxiliary timing exports used by multiple notebooks. - `functions/`: the core Python codebase. This is where reusable loaders, selection rules, decoding helpers, coding-direction code, movement-subspace logic, statistics, and figure helpers live. - `processed_data_code/`: notebooks that read the raw data and generate the intermediate processed outputs used later in the project. - `processed_data/`: saved intermediate results. Most figure notebooks load files from here instead of re-reading all raw NWB sessions. - `Main_figures_codes/`: notebooks that build the main manuscript figures from `processed_data/`. - `Main_figures_pdf/`: exported PDF outputs for the main figures. - `Supplementary_figures_codes/`: notebooks that build the supplementary figures from `processed_data/`. - `Supplementary_figures_pdf/`: exported PDF outputs for the supplementary figures. - `new analysis/`: exploratory or newly structured analyses that are not part of the historical main/supplementary figure pipeline. - `tests/`: lightweight Python tests that verify core helpers such as paths, loading, selection rules, and signal utilities. - `install_dependencies_one_click.ipynb`: the intended entry point for installing Python dependencies for the notebooks. - `requirements.txt`: the package list used by the installation notebook. - `README.md`: project overview, workflow summary, and orientation guide. Inside `functions/`, the files are easiest to read in a few broad groups: - core utilities such as `paths.py`, `core.py`, and `constants.py` - loading code such as `loading_base.py`, `loading_nwb.py`, and `loading_processed.py` - main analysis code such as `psth_build.py`, `decoding_pipelines.py`, `coding_direction.py`, `movement.py`, and `dropout.py` - session-aware helpers such as `session_io.py`, `session_trials.py`, and `session_context.py` - figure helpers such as `figure_psth.py`, `figure_decoding.py`, and `figure_brainmaps.py` Some files such as `function.py`, `loading.py`, `math_utils.py`, `decoding.py`, `pipelines.py`, and `session_aware.py` are mainly compatibility wrappers. They preserve older import paths while the real implementation lives in the smaller focused modules listed above. ## Main Analysis Components Representative modules in `functions/`: - `psth_build.py`: builds canonical per-session, per-area PSTH entries from NWB files. - `loading_processed.py`: canonicalizes saved processed outputs into stable Python structures. - `selection.py`: shared trial and unit selection logic. - `decoding_pipelines.py`: decoding analyses such as delay, pre-whisker training, after-whisker, and neuron-count decoding. - `coding_direction.py`: context, stimulus, and lick coding-direction analyses. - `movement.py` and `subspace.py`: potent/null movement subspace definition and decoding. - `figure_*.py`: reusable helpers for main and supplementary figure notebooks. ## Dependency Installation Use `install_dependencies_one_click.ipynb` from the project root to install the Python dependencies used by the notebooks. This notebook is the intended installation entry point for the project: 1. open `install_dependencies_one_click.ipynb` 2. run its cells from top to bottom 3. let it locate `requirements.txt` and install the required packages The `requirements.txt` file covers the active Python stack used by the repository, including NWB I/O, plotting, decoding, clustering, and movement-subspace dependencies such as `joblib`, `autograd`, and `pymanopt`. After that, the analysis and figure notebooks can be run with the same Python/Jupyter environment. ## Quickstart For a concrete first run: 1. Open `install_dependencies_one_click.ipynb` and run its cells to install the Python environment used by the notebooks. 2. Open `RUN_ALL/run_all_processed_data_code.ipynb` and run it from top to bottom to populate `processed_data/` in dependency order. 3. Open `Main_figures_codes/ghaderi2025_figure1_c_performance_structured.ipynb` and run it to generate or update a figure PDF in `Main_figures_pdf/`. If `processed_data/` is already populated, you can usually skip step 2 and go straight to a figure notebook. ## `RUN_ALL/` Notebooks The `RUN_ALL/` folder contains three orchestrator notebooks: - `RUN_ALL/run_all_processed_data_code.ipynb`: runs the full `processed_data_code/` pipeline in dependency order and writes reports to `RUN_ALL/execution_reports/` - `RUN_ALL/run_all_main_figures_codes.ipynb`: runs all notebooks in `Main_figures_codes/` and writes reports to `RUN_ALL/main_figures_execution_reports/` - `RUN_ALL/run_all_supplementary_figures_codes.ipynb`: runs all notebooks in `Supplementary_figures_codes/` and writes reports to `RUN_ALL/supplementary_figures_execution_reports/` ## Recommended Workflow For most users, the standard workflow is: 1. Run `RUN_ALL/run_all_processed_data_code.ipynb` to build or refresh `processed_data/`. 2. Run `RUN_ALL/run_all_main_figures_codes.ipynb` to regenerate the main figure PDFs. 3. Run `RUN_ALL/run_all_supplementary_figures_codes.ipynb` to regenerate the supplementary figure PDFs. If `processed_data/` is already populated, you can usually skip step 1 and go directly to one of the figure runners. The processed-data runner: - executes the notebooks in a dependency-safe order - uses the repository root as the working directory so relative paths stay consistent - stops on the first execution error by default - can optionally save executed notebook copies for inspection - writes an execution report to `RUN_ALL/execution_reports/` That execution report is a JSON summary of the run. It records: - the repository root and working directory used for execution - the timestamp of the run - the full notebook order that was attempted - whether executed or failed notebook copies were saved - one result entry per notebook with its status, runtime, optional saved copy path, and any error type or error message If you prefer to run the pipeline manually, the broad order is: 1. Build PSTH entries with `processed_data_code/psth_10ms.ipynb` and/or `processed_data_code/psth_5ms.ipynb`. 2. Run the remaining notebooks in `processed_data_code/` to populate `processed_data/`. 3. Run notebooks in `Main_figures_codes/` and `Supplementary_figures_codes/` to regenerate the PDF figures. ## Outputs Typical processed outputs include: - canonical PSTH bundles - decoding accuracy matrices - coding-direction projections - ROC and modulation summaries - movement subspace projections - PCA and temporal-correlation results These outputs are saved under `processed_data/` and are used by the figure notebooks to generate the final manuscript panels. ## Why `.npz` Files - `.npz` is the reference file format for the active Python workflow. - It can be loaded directly with NumPy, which keeps the notebooks simpler and more reliable. - It stores arrays and structured analysis results in a format that fits the Python codebase naturally. - The current Python notebooks are built around `.npz` outputs rather than `.mat` files. ## Current Limitations - The project is notebook-driven; most orchestration still happens through notebooks rather than command-line scripts. - The automated tests are lightweight and focus on core helpers rather than full notebook execution. - The repository contains large binary datasets and precomputed outputs, so cloning and sharing can be heavy. ## Suggested Entry Points For new readers, a practical order is: 1. `functions/` 2. `processed_data_code/` 3. `processed_data/` 4. `Main_figures_codes/` 5. `Supplementary_figures_codes/` This order matches the logic of the pipeline from reusable methods to saved results to final figures.