Reinforcement learning and Bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal

During adolescence, youth venture out, explore the wider world, and are challenged to learn how to navigate novel and uncertain environments. We investigated how performance changes across adolescent development in a stochastic, volatile reversal-learning task that uniquely taxes the balance of persistence and flexibility. In a sample of 291 participants aged 8–30, we found that in the mid-teen years, adolescents outperformed both younger and older participants. We developed two independent cognitive models, based on Reinforcement learning (RL) and Bayesian inference (BI). The RL parameter for learning from negative outcomes and the BI parameters specifying participants’ mental models were closest to optimal in mid-teen adolescents, suggesting a central role in adolescent cognitive processing. By contrast, persistence and noise parameters improved monotonically with age. We distilled the insights of RL and BI using principal component analysis and found that three shared components interacted to form the adolescent performance peak: adult-like behavioral quality, child-like time scales, and developmentally-unique processing of positive feedback. This research highlights adolescence as a neurodevelopmental window that can create performance advantages in volatile and uncertain environments. It also shows how detailed insights can be gleaned by using cognitive models in new ways.

Maria K. Eckstein, Sarah L. Master, Ronald E. Dahl, Linda Wilbrecht, Anne G.E. Collins, Reinforcement learning and Bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal, Developmental Cognitive Neuroscience, Volume 55, 2022, 101106, ISSN 1878-9293, https://doi.org/10.1016/j.dcn.2022.101106, https://www.sciencedirect.com/science/article/pii/S1878929322000494

Reinforcement learning and Bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal2022-06-18T20:50:11+00:00

Making sense of strengths and weaknesses observed in adolescent laboratory rodents

During adolescence, rodents disperse from their natal site, find a new home, and navigate social relationships and threats. Although rats and mice in the laboratory cannot fully express these natural behaviors, they show striking changes in their affective and cognitive behavior across the adolescent period. In some laboratory-based behavior metrics, adolescent rodents fail to show the same behaviors expressed by adults, but in other metrics, adolescent behavioral performance is more robust or more flexible than at other ages. These data are often interpreted in light of proximate level analysis of development of neural circuits. It is also informative to attempt ultimate-level explanations and consider how sex and species-specific adolescent behavioral changes support dispersal, foraging, and social interactions in the wild.

Wan Chen Lin, Linda Wilbrecht, Making sense of strengths and weaknesses observed in adolescent laboratory rodents, Current Opinion in Psychology, Volume 45, 2022, 101297, ISSN 2352-250X, https://doi.org/10.1016/j.copsyc.2021.12.009, https://www.sciencedirect.com/science/article/pii/S2352250X21002499

Making sense of strengths and weaknesses observed in adolescent laboratory rodents2022-06-18T20:42:23+00:00

Sex Differences in Pubertal Circadian and Ultradian Rhythmic Development Under Semi-Naturalistic Conditions

Biological rhythms in core body temperature (CBT) provide informative markers of adolescent development under controlled laboratory conditions. However, it is unknown whether these markers are preserved under more variable, semi-naturalistic conditions, and whether CBT may therefore prove useful in a real-world setting. To evaluate this possibility, we examined fecal steroid concentrations and CBT rhythms from pre-adolescence (p26) through early adulthood (p76) in intact male and female Wistar rats under natural light and climate at the Stephen Glickman Field Station for the Study of Behavior, Ecology and Reproduction. Despite greater environmental variability, CBT markers of pubertal onset and its rhythmic progression were comparable with those previously reported in laboratory conditions in female rats and extend actigraphy-based findings in males. Specifically, sex differences emerged in CBT circadian rhythm (CR) power and amplitude prior to pubertal onset and persisted into early adulthood, with females exhibiting elevated CBT and decreased CR power compared with males. Within-day (ultradian rhythm [UR]) patterns also exhibited a pronounced sex difference associated with estrous cyclicity. Pubertal onset, defined by vaginal opening, preputial separation, and sex steroid concentrations, occurred later than previously reported under lab conditions for both sexes. Vaginal opening and increased fecal estradiol concentrations were closely tied to the commencement of 4-day oscillations in CBT and UR power. By contrast, preputial separation and the first rise in testosterone concentration were not associated with adolescent changes to CBT rhythms in male rats. Together, males and females exhibited unique temporal patterning of CBT and sex steroids across pubertal development, with tractable associations between hormonal concentrations, external development, and temporal structure in females. The preservation of these features outside the laboratory supports CBT as a strong candidate for translational pubertal monitoring under semi-naturalistic conditions in females.

Grant, Azure D., Linda Wilbrecht, and Lance J. Kriegsfeld,  Sex Differences in Pubertal Circadian and Ultradian Rhythmic Development Under Semi-Naturalistic Conditions, Journal of Biological Rhythms, (May 2022). https://doi.org/10.1177/07487304221092715

Sex Differences in Pubertal Circadian and Ultradian Rhythmic Development Under Semi-Naturalistic Conditions2022-06-18T20:34:48+00:00

Adolescent Development of Biological Rhythms: Estradiol Dependence and Effects of Combined Contraceptives

Purpose Adolescence is a period of continuous development, including the maturation of endogenous rhythms across systems and timescales. Although these dynamic changes are well recognized, their continuous structure and hormonal dependence have not been systematically characterized. Given the well-established link between core body temperature (CBT) and reproductive hormones in adults, we hypothesized that high-resolution CBT can be applied to passively monitor pubertal development and disruption with high fidelity.

Methods To examine this possibility, we used signal processing to investigate the trajectory of CBT rhythms at the within-day (ultradian), daily (circadian), and ovulatory timescales, their dependence on estradiol, and the effects of hormonal contraceptives.

Results Puberty onset was marked by a rise in fecal estradiol (fE2), followed by an elevation in CBT and circadian power. This time period marked the commencement of 4-day rhythmicity in fE2, CBT, and ultradian power marking the onset of the estrous cycle. The rise in circadian amplitude was accelerated by E2 treatment, indicating a role for this hormone in rhythmic development. Contraceptive administration in later adolescence reduced CBT and circadian power and resulted in disruption to 4-day cycles that persisted after discontinuation.

Conclusions Our data reveal with precise temporal resolution how biological rhythms change across adolescence and demonstrate a role for E2 in the emergence and preservation of multiscale rhythmicity. These findings also demonstrate how hormones delivered exogenously in a non-rhythmic pattern can disrupt rhythmic development. These data lay the groundwork for a future in which temperature metrics provide an inexpensive, convenient method for monitoring pubertal maturation and support the development of hormone therapies that better mimic and support human chronobiology.

Azure D. Grant, Linda Wilbrecht, Lance J. Kriegsfeld, Adolescent Development of Biological Rhythms: Estradiol Dependence and Effects of Combined Contraceptives, Frontiers in Physiology, Nov. 5, 2021, https://www.frontiersin.org/articles/10.3389/fphys.2021.752363/full

Adolescent Development of Biological Rhythms: Estradiol Dependence and Effects of Combined Contraceptives2022-06-18T14:39:17+00:00

Modeling changes in probabilistic reinforcement learning during adolescence

In the real world, many relationships between events are uncertain and probabilistic. Uncertainty is also likely to be a more common feature of daily experience for youth because they have less experience to draw from than adults. Some studies suggest probabilistic learning may be inefficient in youths compared to adults, while others suggest it may be more efficient in youths in mid adolescence. Here we used a probabilistic reinforcement learning task to test how youth age 8-17 (N = 187) and adults age 18-30 (N = 110) learn about stable probabilistic contingencies. Performance increased with age through early-twenties, then stabilized. Using hierarchical Bayesian methods to fit computational reinforcement learning models, we show that all participants’ performance was better explained by models in which negative outcomes had minimal to no impact on learning. The performance increase over age was driven by 1) an increase in learning rate (i.e. decrease in integration time scale); 2) a decrease in noisy/exploratory choices. In mid-adolescence age 13-15, salivary testosterone and learning rate were positively related. We discuss our findings in the context of other studies and hypotheses about adolescent brain development.

Liyu Xia, Sarah L. Master, Maria K. Eckstein, Beth Baribault, Ronald E. Dahl, Linda Wilbrecht, Anne Gabrielle Eva Collins, Modeling changes in probabilistic reinforcement learning during adolescence, July 2021, https://doi.org/10.1371/journal.pcbi.1008524

Modeling changes in probabilistic reinforcement learning during adolescence2022-06-18T20:53:19+00:00

Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice

Decision-making circuits are modulated across life stages (e.g. juvenile, adolescent, or adult)—as well as on the shorter timescale of reproductive cycles in females—to meet changing environmental and physiological demands. Ovarian hormonal modulation of relevant neural circuits is a potential mechanism by which behavioral flexibility is regulated in females. Here we examined the influence of prepubertal ovariectomy (pOVX) versus sham surgery on performance in an odor-based multiple choice reversal task. We observed that pOVX females made different types of errors during reversal learning compared to sham surgery controls. Using reinforcement learning models fit to trial-by-trial behavior, we found that pOVX females exhibited lower inverse temperature parameter (β) compared to sham females. These findings suggest that OVX females solve the reversal task using a more exploratory choice policy, whereas sham females use a more exploitative policy prioritizing estimated high value options. To seek a neural correlate of this behavioral difference, we performed whole-cell patch clamp recordings within the dorsomedial striatum (DMS), a region implicated in regulating action selection and explore/exploit choice policy. We found that the intrinsic excitability of dopamine receptor type 2 (D2R) expressing indirect pathway spiny projection neurons (iSPNs) was significantly higher in pOVX females compared to both unmanipulated and sham surgery females. Finally, to test whether mimicking this increase in iSPN excitability could recapitulate the pattern of reversal task behavior observed in pOVX females, we chemogenetically activated DMS D2R(+) neurons within intact female mice. We found that chemogenetic activation increased exploratory choice during reversal, similar to the pattern we observed in pOVX females. Together, these data suggest that pubertal status may influence explore/exploit balance in females via the modulation of iSPN intrinsic excitability within the DMS.

Kristen Delevich, Christopher D. Hall, Linda Wilbrecht, Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice, BioRxiv, https://www.biorxiv.org/content/10.1101/2021.06.01.446609v2, doi: https://doi.org/10.1101/2021.06.01.446609

Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice2021-06-17T13:40:25+00:00

Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice

Decision-making circuits are modulated across life stages (e.g. juvenile, adolescent, or adult)—as well as on the shorter timescale of reproductive cycles in females—to meet changing environmental and physiological demands. Ovarian hormonal modulation of relevant neural circuits is a potential mechanism by which behavioral flexibility is regulated in females. Here we examined the influence of prepubertal ovariectomy (pOVX) versus sham surgery on performance in an odor-based multiple choice reversal task. We observed that pOVX females made different types of errors during reversal learning compared to sham surgery controls. Using reinforcement learning models fit to trial-by-trial behavior, we found that pOVX females exhibited lower inverse temperature parameter (β) compared to sham females. These findings suggest that OVX females solve the reversal task using a more exploratory choice policy, whereas sham females use a more exploitative policy prioritizing estimated high value options. To seek a neural correlate of this behavioral difference, we performed whole-cell patch clamp recordings within the dorsomedial striatum (DMS), a region implicated in regulating action selection and explore/exploit choice policy. We found that the intrinsic excitability of dopamine receptor type 2 (D2R) expressing indirect pathway spiny projection neurons (iSPNs) was significantly higher in pOVX females compared to both unmanipulated and sham surgery females. Finally, to test whether mimicking this increase in iSPN excitability could recapitulate the pattern of reversal task behavior observed in pOVX females, we chemogenetically activated DMS D2R(+) neurons within intact female mice. We found that chemogenetic activation increased exploratory choice during reversal, similar to the pattern we observed in pOVX females. Together, these data suggest that pubertal status may influence explore/exploit balance in females via the modulation of iSPN intrinsic excitability within the DMS.

Kristen Delevich, Christopher D. Hall, Linda Wilbrecht, Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice, BioRxiv, June 1, 2021, https://www.biorxiv.org/content/10.1101/2021.06.01.446609v1
doi: https://doi.org/10.1101/2021.06.01.446609

Prepubertal ovariectomy alters dorsomedial striatum indirect pathway neuron excitability and explore/exploit balance in female mice2022-06-18T20:51:18+00:00

Coming of age in the frontal cortex: The role of puberty in cortical maturation

Across species, adolescence is a period of growing independence that is associated with the maturation of cognitive, social, and affective processing. Reorganization of neural circuits within the frontal cortex is believed to contribute to the emergence of adolescent changes in cognition and behavior. While puberty coincides with adolescence, relatively little is known about which aspects of frontal cortex maturation are driven by pubertal development and gonadal hormones. In this review, we highlight existing work that suggests puberty plays a role in the maturation of specific cell types in the medial prefrontal cortex (mPFC) of rodents, and highlight possible routes by which gonadal hormones influence frontal cortical circuit development.

Kristen Delevich, Madeline Klinger, Nana J.Okada, Linda Wilbrecht, Coming of age in the frontal cortex: The role of puberty in cortical maturation, May 10, 2021, https://www.sciencedirect.com/science/article/pii/S108495212100094X
Coming of age in the frontal cortex: The role of puberty in cortical maturation2021-06-02T16:26:26+00:00

Learning Rates Are Not All the Same: The Interpretation of Computational Model Parameters Depends on the Context

Reinforcement Learning (RL) has revolutionized the cognitive and brain sciences, explaining behavior from simple conditioning to problem solving, across the life span, and anchored in brain function. However, discrepancies in results are increasingly apparent between studies, particularly in the developmental literature. To better understand these, we investigated to which extent parameters generalize between tasks and models, and capture specific and uniquely interpretable (neuro)cognitive processes. 291 participants aged 8-30 years completed three learning tasks in a single session, and were fitted using state-of-the-art RL models. RL decision noise/exploration parameters generalized well between tasks, decreasing between ages 8-17. Learning rates for negative feedback did not generalize, and learning rates for positive feedback showed intermediate generalizability, dependent on task similarity. These findings can explain discrepancies in the existing literature. Future research therefore needs to carefully consider task characteristics when relating findings across studies, and develop strategies to computationally model how context impacts behavior.

Maria K Eckstein, Sarah L Master, Liyu Xia, Ronald E Dahl, Linda Wilbrecht, Anne Gabrielle Eva Collins, Learning Rates Are Not All the Same: The Interpretation of Computational Model Parameters Depends on the Context (May 2021), https://www.biorxiv.org/content/10.1101/2021.05.28.446162v1
doi: https://doi.org/10.1101/2021.05.28.446162

Learning Rates Are Not All the Same: The Interpretation of Computational Model Parameters Depends on the Context2022-06-18T20:51:56+00:00

What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience

Reinforcement learning (RL) is a concept that has been invaluable to research fields including machine learning, neuroscience, and cognitive science. However, what RL entails partly differs between fields, leading to difficulties when interpreting and translating findings.

This paper lays out these differences and zooms in on cognitive (neuro)science, revealing that we often overinterpret RL modeling results, with severe consequences for future research. Specifically, researchers often assume—implicitly—that model parameters generalize between tasks, models, and participant populations, despite overwhelming negative empirical evidence for this assumption. We also often assume that parameters measure specific, unique, and meaningful (neuro)cognitive processes, a concept we call interpretability, for which empirical evidence is also lacking.

We conclude that future computational research needs to pay increased attention to these implicit assumptions when using RL models, and suggest an alternative framework that resolves these issues and allows us to unleash the potential of RL in cognitive (neuro)science.

Maria Eckstein, Linda Wilbrecht, Anne Collins, What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience (May 2021), https://psyarxiv.com/e7kwx/

 

What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience2022-06-18T20:52:15+00:00