Measuring Diverse Gender Identities: Phase 2

WASHINGTON, D.C. — Understanding and giving a voice to the diverse experiences that people have is an important part of social and demographic research. Historically, however, some segments of the population — such as gender minorities — have not been measured and given that voice. Although researchers and federal statistical systems are now working to overcome the data gap for gender minorities, more work is needed.
One challenge for gender research involves creating questions that accurately measure diverse gender identities. To overcome this challenge and accurately measure the experiences of all survey participants, Gallup conducted extensive testing of new, more inclusive and accurate gender questions.
Sometimes a single question capturing broad — but still inclusive — gender identity suffices. Other times, research may require more nuanced gender identity reporting. In the first article, we discussed how we developed a baseline gender measure that is a single, simple question and includes an option other than male or female. This article focuses on the development of more detailed items that can be applied in a variety of research settings.
We tested four different question versions in a web survey experiment of 22,882 U.S. adults, conducted April 3-13, 2023, via the Gallup Panel. Respondents were randomly assigned to answer one of the four question variants.
It was also important to us to include the input of non-cisgender experiences in this research. After the gender question(s), respondents who selected an identity other than male (man) or female (woman) were asked to provide open-ended feedback on the questions they were asked (see footnote for exact wording).1
In the first version, we asked the single, broad gender question that is fielded on many of our U.S. surveys of the general population. This is the version of the question we adopted after our first round of experiments.
What is your gender?
In the second version, we modified the first question to adopt gender terminology (man and woman) versus terms used to describe sex (male and female). A common critique of gender questions, including our version 1 question, is that they ask about gender but use terms for sex in the response options. We wanted to test whether a change in terms would produce a difference in our estimates.
What is your gender?
The third question we tested is the wording recommended by the National Academies in a 2022 Issue Brief on measuring sex, gender identity and sexual orientation. Questions like this are important not only because they present respondents with options that reflect their identity but also provide categories that allow researchers to measure and understand the diversity of gender experiences.
What sex were you assigned at birth, on your original birth certificate?
What is your current gender?
In the fourth version, we modified the National Academies’ wording in five important ways:
What sex were you assigned at birth, on your original birth certificate?
Which of the following best describes your gender? (You may select all that apply.)
When comparing the first and second versions, we found no significant differences in the results. Switching from terms used to describe sex (male and female) to gender (man and woman) did not produce a difference in our estimates.
Next, we compared the results of versions 3 and 4, which were two-part questions asking about sex and gender. Both questions started with a question asking about sex assigned at birth. Both question versions were the same, and as expected, the results of this question were nearly identical for versions 3 and 4.
We then compared the results of the gender questions. Version 3 used terms to describe sex, while version 4 used terms to describe gender. As we saw when comparing versions 1 and 2, we observed no meaningful difference between groups 3 and 4 in the percentage of respondents who reported male/man or female/woman.
We changed version 4 to adopt a “select all that apply” format. This change did not produce any significant difference in the distributions of male/man and female/woman. In version 4, 0.8% of respondents (n=47) selected two options and 0.1% (n=4) selected three options.
In version 3, 0.1% of respondents (n=6) selected the two-spirit option. Version 4 displayed this option to all respondents, regardless of reported race or ethnicity. Of the six respondents who selected the two-spirit option, one identified as Native American.
In version 4, 1.0% of respondents (n=57) selected the nonbinary response option, which was the primary difference observed between version 3 and version 4.
After each question version, we asked all respondents who selected any option(s) other than male/man or female/woman to provide their feedback on the questions. Of the 233 people who were asked the open-ended question, 116 provided a response. An analysis of the qualitative feedback identified several themes:
“Gender identity is a continuum, and some of your questions offer response choices that are binary and deterministic.”
“‘Transgender’” feels like it precludes me from male or female.”
“Transgender is not a gender in and of itself and should not be listed in contrast with male or female….”
“Sex at birth is not relevant to anything other than medical questions.”
“Assigned sex at birth is not an acceptable question in any circumstance, other than maybe certain medical situations.”
From this research, we developed several recommendations. These can be taken into consideration specific to the research objectives and target population of a given study.
Our research included questions about transgender identities, but our results were not conclusive and are not discussed in detail in this article. This component of our research will be shared in a future blog. However, a clear finding is that trans is not in and of itself a gender identity and should not be treated as such in gender questions.
Finally, we want to acknowledge that accurately measuring gender in a survey question is complex. Identities and experiences are diverse, gender is not always binary, and terminology continues to evolve. Our recommendations are intended to be just that — recommendations. There is currently no perfect question that fits all research objectives, but the pursuit of a perfect question shouldn’t prevent researchers from asking about gender using the current understanding of best practices. We hope this research adds to the knowledge and discussion about gender survey questions and that research in this area continues to develop and refine gender questions.
To stay up to date with the latest Gallup News insights and updates, follow us on Twitter.