Learnable latent embeddings for joint behavioral and neural analysis
Abstract
Mapping behavioral actions to neural activity is a fundamental goal of neuroscience. As our ability to record large neural and behavioral data increases, there is growing interest in modeling neural dynamics during adaptive behaviors to probe neural representations. In particular, neural latent embeddings can reveal underlying correlates of behavior, yet, we lack non-linear techniques that can explicitly and flexibly leverage joint behavior and neural data to uncover neural dynamics. Here, we fill this gap with a novel encoding method, CEBRA, that jointly uses behavioral and neural data in a (supervised) hypothesis- or (self-supervised) discovery-driven manner to produce both consistent and high-performance latent spaces. We show that consistency can be used as a metric for uncovering meaningful differences, and the inferred latents can be used for decoding. We validate its accuracy and demonstrate our tool's utility for both calcium and electrophysiology datasets, across sensory and motor tasks, and in simple or complex behaviors across species. It allows for single and multi-session datasets to be leveraged for hypothesis testing or can be used label-free. Lastly, we show that CEBRA can be used for the mapping of space, uncovering complex kinematic features, produces consistent latent spaces across 2-photon and Neuropixels data, and can provide rapid, high-accuracy decoding of natural movies from visual cortex.
CEBRA embedding visualizations and decoding
Application of CEBRA-Behavior to rat hippocampus data (Grosmark and Buzsáki, 2016), showing position/neural activity (left), overlayed with decoding obtained by CEBRA. The current point in embedding space is highlighted (right). CEBRA obtains a median absolute error of 5cm (total track length: 160cm; see pre-print for details). Video is played at 2x real-time speed.
CEBRA applied to mouse primary visual cortex, collected at the Allen Institute (de Vries et al. 2020, Siegle et al. 2021). The left panels show example calcium traces from 2-photon imaging (top) and spikes from Neuropixels recording (bottom) while the video (far right) is presented to mice. The center panel shows an embedding space constructed by jointly training a CEBRA-Behavior model with 2-photon and Neuropixels recordings using DINO frame features as labels. The colored trace is an embedding of a held-out test repeat from Neuropixels data. The colormap indicates frame number of the 30 second long video (30 Hz). The last panels show true video (top) and the predicted frame sequence (bottom) using a kNN decoder on the CEBRA-Behavior embedding from the test set.
Pre-Print
The pre-print is available on arxiv now: arxiv.org/abs/2204.00673.
Software
Our official implementation of the CEBRA algorithm will be released on GitHub upon publication of the paper. Watch and Star the repository to be notified of updates and code release dates. You can also follow us on Twitter or subscribe to our mailing list for updates on the project and release timeline.
If you are interested in collaborations and/or early access, please contact us via email.
BibTeX
Please cite our pre-print as follows:
author = {Schneider, Steffen and Lee, Jin H and Mathis, Mackenzie W},
title = {Learnable latent embeddings for joint behavioral and neural analysis},
doi = {10.48550/ARXIV.2204.00673},
url = {https://arxiv.org/abs/2204.00673},
journal = {CoRR},
volume = {abs/2204.00673},
year = {2022},
}