Modeling changes in probabilistic reinforcement learning during adolescence

In the real world, many relationships between events are uncertain and probabilistic. Uncertainty is also likely to be a more common feature of daily experience for youth because they have less experience to draw from than adults. Some studies suggest probabilistic learning may be inefficient in youths compared to adults, while others suggest it may be more efficient in youths in mid adolescence. Here we used a probabilistic reinforcement learning task to test how youth age 8-17 (N = 187) and adults age 18-30 (N = 110) learn about stable probabilistic contingencies. Performance increased with age through early-twenties, then stabilized. Using hierarchical Bayesian methods to fit computational reinforcement learning models, we show that all participants’ performance was better explained by models in which negative outcomes had minimal to no impact on learning. The performance increase over age was driven by 1) an increase in learning rate (i.e. decrease in integration time scale); 2) a decrease in noisy/exploratory choices. In mid-adolescence age 13-15, salivary testosterone and learning rate were positively related. We discuss our findings in the context of other studies and hypotheses about adolescent brain development.

Liyu Xia, Sarah L. Master, Maria K. Eckstein, Beth Baribault, Ronald E. Dahl, Linda Wilbrecht, Anne Gabrielle Eva Collins, Modeling changes in probabilistic reinforcement learning during adolescence, July 2021, https://doi.org/10.1371/journal.pcbi.1008524

Modeling changes in probabilistic reinforcement learning during adolescence2022-06-18T20:53:19+00:00

Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice

Decision-making circuits are modulated across life stages (e.g. juvenile, adolescent, or adult)—as well as on the shorter timescale of reproductive cycles in females—to meet changing environmental and physiological demands. Ovarian hormonal modulation of relevant neural circuits is a potential mechanism by which behavioral flexibility is regulated in females. Here we examined the influence of prepubertal ovariectomy (pOVX) versus sham surgery on performance in an odor-based multiple choice reversal task. We observed that pOVX females made different types of errors during reversal learning compared to sham surgery controls. Using reinforcement learning models fit to trial-by-trial behavior, we found that pOVX females exhibited lower inverse temperature parameter (β) compared to sham females. These findings suggest that OVX females solve the reversal task using a more exploratory choice policy, whereas sham females use a more exploitative policy prioritizing estimated high value options. To seek a neural correlate of this behavioral difference, we performed whole-cell patch clamp recordings within the dorsomedial striatum (DMS), a region implicated in regulating action selection and explore/exploit choice policy. We found that the intrinsic excitability of dopamine receptor type 2 (D2R) expressing indirect pathway spiny projection neurons (iSPNs) was significantly higher in pOVX females compared to both unmanipulated and sham surgery females. Finally, to test whether mimicking this increase in iSPN excitability could recapitulate the pattern of reversal task behavior observed in pOVX females, we chemogenetically activated DMS D2R(+) neurons within intact female mice. We found that chemogenetic activation increased exploratory choice during reversal, similar to the pattern we observed in pOVX females. Together, these data suggest that pubertal status may influence explore/exploit balance in females via the modulation of iSPN intrinsic excitability within the DMS.

Kristen Delevich, Christopher D. Hall, Linda Wilbrecht, Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice, BioRxiv, https://www.biorxiv.org/content/10.1101/2021.06.01.446609v2, doi: https://doi.org/10.1101/2021.06.01.446609

Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice2021-06-17T13:40:25+00:00

Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice

Decision-making circuits are modulated across life stages (e.g. juvenile, adolescent, or adult)—as well as on the shorter timescale of reproductive cycles in females—to meet changing environmental and physiological demands. Ovarian hormonal modulation of relevant neural circuits is a potential mechanism by which behavioral flexibility is regulated in females. Here we examined the influence of prepubertal ovariectomy (pOVX) versus sham surgery on performance in an odor-based multiple choice reversal task. We observed that pOVX females made different types of errors during reversal learning compared to sham surgery controls. Using reinforcement learning models fit to trial-by-trial behavior, we found that pOVX females exhibited lower inverse temperature parameter (β) compared to sham females. These findings suggest that OVX females solve the reversal task using a more exploratory choice policy, whereas sham females use a more exploitative policy prioritizing estimated high value options. To seek a neural correlate of this behavioral difference, we performed whole-cell patch clamp recordings within the dorsomedial striatum (DMS), a region implicated in regulating action selection and explore/exploit choice policy. We found that the intrinsic excitability of dopamine receptor type 2 (D2R) expressing indirect pathway spiny projection neurons (iSPNs) was significantly higher in pOVX females compared to both unmanipulated and sham surgery females. Finally, to test whether mimicking this increase in iSPN excitability could recapitulate the pattern of reversal task behavior observed in pOVX females, we chemogenetically activated DMS D2R(+) neurons within intact female mice. We found that chemogenetic activation increased exploratory choice during reversal, similar to the pattern we observed in pOVX females. Together, these data suggest that pubertal status may influence explore/exploit balance in females via the modulation of iSPN intrinsic excitability within the DMS.

Kristen Delevich, Christopher D. Hall, Linda Wilbrecht, Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice, BioRxiv, June 1, 2021, https://www.biorxiv.org/content/10.1101/2021.06.01.446609v1
doi: https://doi.org/10.1101/2021.06.01.446609

Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice2022-07-29T03:12:19+00:00

Coming of age in the frontal cortex: The role of puberty in cortical maturation

Across species, adolescence is a period of growing independence that is associated with the maturation of cognitive, social, and affective processing. Reorganization of neural circuits within the frontal cortex is believed to contribute to the emergence of adolescent changes in cognition and behavior. While puberty coincides with adolescence, relatively little is known about which aspects of frontal cortex maturation are driven by pubertal development and gonadal hormones. In this review, we highlight existing work that suggests puberty plays a role in the maturation of specific cell types in the medial prefrontal cortex (mPFC) of rodents, and highlight possible routes by which gonadal hormones influence frontal cortical circuit development.

Kristen Delevich, Madeline Klinger, Nana J.Okada, Linda Wilbrecht, Coming of age in the frontal cortex: The role of puberty in cortical maturation, May 10, 2021, https://www.sciencedirect.com/science/article/pii/S108495212100094X
Coming of age in the frontal cortex: The role of puberty in cortical maturation2021-06-02T16:26:26+00:00

Learning Rates Are Not All the Same: The Interpretation of Computational Model Parameters Depends on the Context

Reinforcement Learning (RL) has revolutionized the cognitive and brain sciences, explaining behavior from simple conditioning to problem solving, across the life span, and anchored in brain function. However, discrepancies in results are increasingly apparent between studies, particularly in the developmental literature. To better understand these, we investigated to which extent parameters generalize between tasks and models, and capture specific and uniquely interpretable (neuro)cognitive processes. 291 participants aged 8-30 years completed three learning tasks in a single session, and were fitted using state-of-the-art RL models. RL decision noise/exploration parameters generalized well between tasks, decreasing between ages 8-17. Learning rates for negative feedback did not generalize, and learning rates for positive feedback showed intermediate generalizability, dependent on task similarity. These findings can explain discrepancies in the existing literature. Future research therefore needs to carefully consider task characteristics when relating findings across studies, and develop strategies to computationally model how context impacts behavior.

Maria K Eckstein, Sarah L Master, Liyu Xia, Ronald E Dahl, Linda Wilbrecht, Anne Gabrielle Eva Collins, Learning Rates Are Not All the Same: The Interpretation of Computational Model Parameters Depends on the Context (May 2021), https://www.biorxiv.org/content/10.1101/2021.05.28.446162v1
doi: https://doi.org/10.1101/2021.05.28.446162

Learning Rates Are Not All the Same: The Interpretation of Computational Model Parameters Depends on the Context2022-06-18T20:51:56+00:00

What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience

Reinforcement learning (RL) is a concept that has been invaluable to research fields including machine learning, neuroscience, and cognitive science. However, what RL entails partly differs between fields, leading to difficulties when interpreting and translating findings.

This paper lays out these differences and zooms in on cognitive (neuro)science, revealing that we often overinterpret RL modeling results, with severe consequences for future research. Specifically, researchers often assume—implicitly—that model parameters generalize between tasks, models, and participant populations, despite overwhelming negative empirical evidence for this assumption. We also often assume that parameters measure specific, unique, and meaningful (neuro)cognitive processes, a concept we call interpretability, for which empirical evidence is also lacking.

We conclude that future computational research needs to pay increased attention to these implicit assumptions when using RL models, and suggest an alternative framework that resolves these issues and allows us to unleash the potential of RL in cognitive (neuro)science.

Maria Eckstein, Linda Wilbrecht, Anne Collins, What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience (May 2021), https://psyarxiv.com/e7kwx/

 

What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience2022-06-18T20:52:15+00:00

The Unique Advantage of Adolescents in Probabilistic Reversal: Reinforcement Learning and Bayesian Inference Provide Adequate and Complementary Models

During adolescence, youth venture out, explore the wider world, and are challenged to learn how to navigate novel and uncertain environments. We investigated whether adolescents are uniquely adapted to this transition, compared to younger children and adults. In a stochastic, volatile reversal learning task with a sample of 291 participants aged 8-30, we found that adolescents 13-15 years old outperformed both younger and older participants. We developed two independent cognitive models, one based on Reinforcement learning (RL) and the other Bayesian inference (BI), and used hierarchical Bayesian model fitting to assess developmental changes in underlying cognitive mechanisms. Choice parameters in both models improved monotonously. By contrast, RL update parameters and BI mental-model parameters peaked closest to optimal values in 13-to-15-year-olds. Combining both models using principal component analysis yielded new insights, revealing that three readily-interpretable components contributed to the early-to mid-adolescent performance peak. This research highlights early-to mid-adolescence as a neurodevelopmental window that may be more optimal for behavioral adjustment in volatile and uncertain environments. It also shows how increasingly detailed insights can be gleaned by invoking different cognitive models.

Maria K. Eckstein, Sarah L. Master, Ronald E. Dahl, Linda Wilbrecht, Anne G.E. Collins, The Unique Advantage of Adolescents in Probabilistic Reversal: Reinforcement Learning and Bayesian Inference Provide Adequate and Complementary Models (Mar. 2021) https://www.biorxiv.org/content/10.1101/2020.07.04.187971v2.full
doi: https://doi.org/10.1101/2020.07.04.187971

The Unique Advantage of Adolescents in Probabilistic Reversal: Reinforcement Learning and Bayesian Inference Provide Adequate and Complementary Models2022-06-18T20:52:28+00:00

A role for adaptive developmental plasticity in learning and decision making

From both a medical and educational perspective, there is enormous value to understanding the environmental factors that sculpt learning and decision making. These questions are often approached from proximate levels of analysis, but may be further informed by the adaptive developmental plasticity framework used in evolutionary biology. The basic adaptive developmental plasticity framework posits that biological sensitive periods evolved to use information from the environment to sculpt emerging phenotypes. Here, we lay out how we can apply this framework to learning and decision making in the mammalian brain and propose a working model in which dopamine neurons and their activity may serve to inform downstream circuits about environmental statistics. More widespread use of this evolutionary framework and its associated models can help inform and guide basic research and intervention science.

Wan Chen Lin, Kristen Delevich, Linda Wilbrecht, A role for adaptive developmental plasticity in learning and decision making, 36 Current Opinion in Behavioral Sciences pp 48–54 (2020), https://doi.org/10.1016/j.cobeha.2020.07.010, http://www.sciencedirect.com/science/article/pii/S2352154620301121

A role for adaptive developmental plasticity in learning and decision making2021-06-02T16:28:40+00:00

Choice suppression is achieved through opponent but not independent function of the striatal indirect pathway in mice

The dorsomedial striatum (DMS) plays a key role in action selection, but little is known about how direct and indirect pathway spiny projection neurons (dSPNs and iSPNs) contribute to choice suppression in freely moving animals. Here, we used pathway-specific chemogenetic manipulation during a serial choice foraging task to test opposing predictions for iSPN function generated by two theories: 1) the ‘select/suppress’ heuristic which suggests iSPN activity is required to suppress alternate choices and 2) the network-inspired Opponent Actor Learning model (OpAL) which proposes that the weighted difference of dSPN and iSPN activity determines choice. We found that chemogenetic activation, but not inhibition, of iSPNs disrupted learned suppression of nonrewarded choices, consistent with the predictions of the OpAL model. Our findings suggest that iSPNs’ role in stopping and freezing does not extend in a simple fashion to choice suppression. These data may provide insights critical for the successful design of interventions for addiction or other conditions in which suppression of behavior is desirable.

Kristen Delevich, Benjamin Hoshal, Anne GE Collins, Linda Wilbrecht, Choice suppression is achieved through opponent but not independent function of the striatal indirect pathway in mice, BioRxiv, https://www.biorxiv.org/content/10.1101/675850v3
doi: https://doi.org/10.1101/675850

 

Choice suppression is achieved through opponent but not independent function of the striatal indirect pathway in mice2022-06-18T20:52:41+00:00

Sex and Pubertal Status Influence Dendritic Spine Density on Frontal Corticostriatal Projection Neurons in Mice

In humans, nonhuman primates, and rodents, the frontal cortices exhibit grey matter thinning and dendritic spine pruning that extends into adolescence. This maturation is believed to support higher cognition but may also confer psychiatric vulnerability during adolescence. Currently, little is known about how specific cell types in the frontal cortex mature or whether puberty plays a role in the maturation of some cell types but not others. Here, we used mice to characterize the spatial topography and adolescent development of cross-corticostriatal (cSTR) neurons that project through the corpus collosum to the dorsomedial striatum. We found that apical spine density on cSTR neurons in the medial prefrontal cortex decreased significantly between late juvenile (P29) and young adult time points (P60), with females exhibiting higher spine density than males at both ages. Adult males castrated prior to puberty onset had higher spine density compared to sham controls. Adult females ovariectomized before puberty onset showed greater variance in spine density measures on cSTR cells compared to controls, but their mean spine density did not significantly differ from sham controls. Our findings reveal that these cSTR neurons, a subtype of the broader class of intratelencephalic-type neurons, exhibit significant sex differences and suggest that spine pruning on cSTR neurons is regulated by puberty in male mice.

Kristen Delevich, Nana J Okada, Ameet Rahane, Zicheng Zhang, Christopher D Hall, Linda Wilbrecht, Sex and Pubertal Status Influence Dendritic Spine Density on Frontal Corticostriatal Projection Neurons in Mice, Cerebral Cortex, , bhz325, https://doi.org/10.1093/cercor/bhz325 (preprint available at https://www.biorxiv.org/content/biorxiv/early/2019/09/30/787408.full.pdf)

Sex and Pubertal Status Influence Dendritic Spine Density on Frontal Corticostriatal Projection Neurons in Mice2020-02-13T03:35:21+00:00

This site does not reflect the views of the University of California and should not be attributed to the University.