The limitations of scientific inquiry

science May 25, 2019

While modern marvels testify of the power of the scientific process, science has major limitations, especially when it attempts to describe social and psychological processes. These limitations become large barriers whenever a particular question becomes highly politicized.

Physics and other similar sciences have a great tool a their disposal - the experiment. They can take physical materials, perform actions on said materials in specific, pre-planned ways, and get results that can be easily used to show cause and effect to prove or disprove a hypothesis.

The social sciences are much more limited. It often is either unethical or impossible to perform experiments of the kind that would be needed to show cause and effect in the psychological or sociological sciences. For example, researchers cannot randomly assign newborn children to different families to test a hypothesis about how certain family situations affect outcomes they might be interested in.

The lack of the ability to do true experiments doesn't mean no research can be done, but rather that alternate ways to study phenomena must be found. Various methods are used: cross-sectional and longitudinal studies that draw on representative, random samples of the population; natural experiments where unrelated phenomena assign individuals to treatment and non-treatment groups; and surveys of individuals recruited to participate in studies.

Not all study designs are equal in their ability to discover truths about the larger population. Whenever a representative sample cannot be drawn for the study, it can only be used to describe the people who participated in the study, but cannot be generalized. For example, consider the following photo:

A jar with red and white gems, and another jar with only blue gems

Let's say each gem in the jars represents an individual, with the colors representing some outcome of a condition a researcher wishes to study. Now suppose that for some reason, the researcher, whether by choice or accident, can only select gems from the jar holding red and white gems. The results would look like the following image:

Red and white gems

It would be pretty easy to conclude that only white and red gems exist in the population. But if a true random sample of the entire population were drawn, the results would look more like the following:

Red, blue, and white gems

With a properly-drawn random sample from the entire population, rather than a subset, we have a much clearer picture and can see that red, white, and blue gems exist.

It turns out that the bad example in the above scenario can be an easy trap to fall into in the social sciences, especially when an issue becomes politicized.

Let's suppose that the gems in the jars above represent the entire population of people who have experienced same-sex attraction. Each color represents some experience and outcome of what a researcher chooses to term "conversion therapy". The blue gems represent people who report having experienced "conversion therapy" who say that it helped them change to some degree. The white gems represent people who report never having experienced "conversion therapy". The red gems represent people who report having experienced varying amounts of harm from "conversion therapy". If the researcher draws only from the group with red and white gems, or even from a group that has some blue gems but a proportion that is not equivalent to the actual population, the researcher can only draw conclusions about the people in the sample they drew, not about the population as a whole.

Equality Utah is basing much of their argument for increased suicidality and depression as a result of so-called "conversion therapy" on a study that has this precise weakness. It includes 245 participants who "self-identified as LGBT" and were recruited from "local bars, clubs, and community agencies that serve [the] population in a 100-mile radius of the research center".

What does this method of sampling tell us about the results of the survey? For one, we can conclude that it does not adequately represent anyone who has experienced therapy, as it does not include anyone who does not currently identify as LGBT. For one, someone like me would not appear in any study following this methodology as I have never identified as gay. Also, we can conclude that the sample is much more representative of people who choose to live in the area where the researcher conducted her experiment, since it only includes people living near the research center. As people's approaches to same-gender attraction vary widely across the nation, this sample is biased in favor of the ideologies practiced near the research center. These are near-fatal problems for the study, at least if it is to be used to indicate what a general population might experience with therapy.

Other problems also exist with the study. Only two questions were asked relating to Sexual Orientation Change Efforts (SOCE):

Participants responded to two items that assessed past parental and caregiver-initiated efforts to change the youths’ sexual orientation. The first item asked: “Between ages 13 and 19, how often did any of your parents/caregivers try to change your sexual orientation (i.e., to make you straight)?” (0 = never [49.64%]; 1 = ever [53.06%]). A second item asked: “Between ages 13 and 19, how often did any of your parents/caregivers take you to a therapist or religious leader to cure, treat, or change your sexual orientation?” (0 = never [65.71%]; 1 = ever [34.29%]). We created a single measure with three categories that identifies the severity of parent-initiated attempts to change youths’ sexual orientation. The three categories include: (1) no attempt to change sexual orientation (n = 109; 41.63%), (2) parent and caregiver-initiated attempt to change sexual orientation without external conversion efforts (n = 52; 21.22%), and (3) parent and caregiver-initiated attempt to change sexual orientation with external conversion efforts (n = 78; 31.84%). Six participants who reported conversion efforts but not parental attempts to change sexual orientation were dropped from the current study, for a total analytic sample of 239 participants.

Can you spot the problems here? First, the survey lumps the efforts of untrained individuals ("religious leader") with licensed therapists. In addition, there is no information as to what types of therapies were attempted. Given that there are many different ways to attempt SOCE, a study that lumps all of these together and further confuses the situation with parent and religious leader attempts can really tell us little about what effect various approaches to SOCE conducted by a trained, licensed therapist would have on youth.

There certainly are some ways of attempting SOCE that can reasonably be assumed to cause harm. However, most modern SOCE is simply talk therapy that does not guarantee change, but explores what possibility there might be of change, and, for the most part, does not even focus on "change" itself, but rather on understanding emotions, learning to define and hold boundaries, and working through past trauma. My own therapy was mostly spent on boundary setting and understanding emotions. It also explored alternate interpretations of what my feelings meant, and what "attractions" really meant. The vast majority of this exploration was done by myself, not at the suggestion of my therapist, as a result of learning to understand my emotional responses better. I also learned to explore my stories around my experiences in order to really understand what they meant for me rather than accepting my initial, societally conditioned, interpretation of said stories.

To really understand what effects SOCE may have on youth, all forms must be evaluated on their own, rather than being lumped together. In addition, they should be separated from the efforts parents and untrained individuals engage in to help youth.