How Mice Decide To Make Love or War

Dog owners whose pets meet during a walk are familiar with the immediate sniffing investigation that typically ensues. Initially, the owners cannot tell whether their dogs will wind up fighting, playing, or trying to mount each other. Something is clearly happening in the dog's brain to make it decide how to behave toward the other dog—but what is going on?

A new study from Caltech examines this question in mice: namely, how does a male mouse sniffing a newly encountered fellow mouse decide whether to make love or war—or to do neither and just mind its own business? The research reveals the neural circuitry that connects olfactory information about another mouse's sex to decision-making points in the mouse brain that determine its behavior.

The study was led by postdoctoral scholars Bin Yang (PhD '22) and Tomomi Karigo, and conducted in the laboratory of David Anderson, Seymour Benzer Professor of Biology, Tianqiao and Chrissy Chen Institute for Neuroscience Leadership Chair, Howard Hughes Medical Institute Investigator, and director of the Tianqiao and Chrissy Chen Institute for Neuroscience. A paper describing the findings appeared online in the journal Nature on August 3.

"Understanding how a mouse chooses whether to mate or fight with a fellow mouse represents decision-making in the brain at its most basic level, and can serve as a model for more complex decision-making processes, even in our own brains," Anderson says.

When encountering another mouse, the male mouse's brain has to decide how to behave by answering two questions about the new animal: "What is it?", and "What should I do about it?" Answering these questions requires that the brain decode the sex identity of the other animal and transform that sex code into a plan of action.

This process occurs as electrical activity—triggered in the mouse's nose by the smell of a male or female—flows into the brain through a series of structures, or nodes, until reaching a decision point that controls the choice of behavior: mating or fighting. The problem is to understand what each of these nodes is doing during this process, and how they perform their function.

It was previously known that the initial decoding of the sex identity of another mouse occurs in a node called the medial amygdala, which receives input from the olfactory system. It was also known that further "downstream" in the circuit, mating or fighting behavior are controlled by two nodes in the hypothalamus called the MPOA and VMHvl, respectively. But in between the amygdala and the hypothalamus lies an enigmatic node called the "BNST" (Bed Nucleus of the Stria Terminalis). What does the BNST do, and how does it do it?

Earlier studies showed that if neurons in the BNST are killed or electrically silenced, male mice encountering a female fail to transition from sniffing to mounting, while those encountering a male fail to transition from sniffing to attack. These findings suggested that BNST serves as a kind of gate that controls whether the sex identity of the mouse (initially decoded by the amygdala) is used to drive the initiation of mounting or attack. But how exactly is that brain "gate" opened?

To answer this question, Yang and the team visualized neuronal activity in BNST by using a miniature microscope attached to a male mouse's head while it interacted with another mouse (either a male or a female). The male mouse has been genetically engineered so that individual neurons glow with light when activated, and the microscope detects these pinpoint flashes. The question is which neurons are active, and what do they do? The team found that there are two types of neurons in the BNST: those that respond preferentially to female mice (female tuned), and those that respond preferentially to males (male tuned). Interestingly, the female-tuned neurons outnumbered the male-tuned neurons in BNST by almost two to one, like those in the amygdala. This indicated that the coding of sex identity by the amygdala was relayed to the BNST and mapped onto its neurons.

The predominance of female-responsive neurons is also observed in MPOA, the structure that lies downstream of BNST and controls male mating behavior. Evidently, the entire circuit appears wired to preferentially respond to females. The one exception to this rule was VMHvl, the node that controls intermale aggression. In this structure, male-tuned neurons outnumber female-tuned neurons by about two to one—the opposite of what is seen in the other circuit nodes. Apparently, VMHvl is like an island of neurons dominated by male responses in a sea of surrounding circuitry otherwise dominated by female responses.

To understand how this inversion of sex-tuning dominance occurs in VMHvl, Yang investigated next how patterns of neuronal activity in this node were altered by silencing neurons in BNST. Surprisingly, he found that when BNST neurons were turned off, the dominance of male-tuned neurons in VMHvl was flipped to a female-dominant response like that observed in MPOA and BNST. This may explain why silencing BNST blocks the transition to aggression: there are no longer enough male-tuned neurons in VMHvl to activate the region sufficiently to produce attack.

By contrast, in the case of MPOA, there was no obvious change in the ratio of female-tuned to male-tuned neurons when BNST neurons were silenced. However, close inspection of neuronal activity in MPOA during interactions with a female revealed that as males transitioned from sniffing to mounting, different neurons were activated in a sequence. During sniffing, one population of neurons was active, but as the animals began to mount, that population was switched off, and a different population of MPOA neurons became active. When BNST neurons were silenced, however, the "sniffing" neurons continued to be active and the "mounting" neurons never turned on.

Thus, BNST neurons are required to "open" a neural gate that allows the transition from sniffing to mounting (toward a female), or to attack (toward a male). Unexpectedly, however, it controls this gate using different mechanisms for attack versus mounting. In the case of attack, BNST operates via quantity control: it ensures that a sufficient number of male-selective neurons are active in VMHvl to reach a threshold for attack. In the case of mounting, BNST operates via quality control: it ensures that sniff-tuned neurons in MPOA are switched off and replaced by mount-tuned neurons.

"These studies begin to shed light on the fundamental question of how the brain transforms a neural representation of object identity into a decision to execute a particular behavior," Yang says. "Future studies should help to uncover how this transformation is implemented at the level of specific synapses."

Reference: Yang B, Karigo T, Anderson DJ. Transformations of neural representations in a social behaviour network. Nature. 2022;608(7924):741-749. doi: 10.1038/s41586-022-05057-6

This article has been republished from the following materials. Note: material may have been edited for length and content. For further information, please contact the cited source.