Ethics of AI Perpetuating Racial Inequalities in Healthcare Delivery and Patient Outcomes

Abstract

Disproportionately high rates of mortality and severe disease experienced by African Americans during this COVID-19 pandemic have been reported by the Centers for Disease Control and Prevention (CDC). Initial characteristic data from 14 states identified 18 percent of overall COVID-19 patients and 33 percent of all those hospitalized from March 1 to March 28 as black.[i] However, this report is missing data from the majority of states. Policymakers and medical experts have called for states and cities to provide greater transparency and more demographic data on patients’ racial and ethnic profiles.[ii] The COVID-19 racial disparity highlights a growing problem: Artificial intelligence (AI) may play a role in deepening racial health disparities in this country. AI is increasingly being utilized in healthcare settings to improve patient outcomes by using big data to inform clinical decision making. Like in many industries outside of healthcare, AI has the potential to revolutionize the current healthcare delivery system. The technological paradigm shift creates concerns with the gender and racial inequalities that these advancements may perpetuate. Specifically, I am troubled by the lack of oversight in formulating the algorithms used in AI and the sources and quality of the data sets. This essay focuses on the potential of AI to further entrench racial disparities in healthcare delivery and patient outcomes throughout healthcare institutions in this country.

Racial biases have historically pervaded medical education and research institutions and are perpetuated by systemic racism. However, there is a shift toward improving “structural competency” to include social and economic influences on healthcare decision making in an effort to broaden our approach to race and culture; this should be extended to the development of AI applications in healthcare. Racial biases have impacted medical treatment and research as well as the code formulation for predictor algorithms already being used in health systems. Understanding the biases that would skew AI, like narrow, unrepresentative data with a lack of female and minority research participants, should be important to all stakeholders.

The significant consequences and risks of systemic racism in medicine are often overlooked. For example, most commercial spirometers, the tool used to assess lung capacity and diagnose respiratory diseases, have “race correction” built into their software. This demonstrates the fraught history of racism in healthcare and unconscious bias in modern healthcare delivery. Thomas Jefferson, in the slavery-based republic, was the first to speculate about racial differences in lung capacity. Samuel Cartwright, who in addition to being a physician, was a plantation owner and slaveholder, built his own spirometer with the support of Thomas Jefferson. Benjamin Gould, also supported by Thomas Jefferson, performed a large anthropometric survey during the Civil War and dedicated an entire chapter to lung capacity of soldiers in the Civil War based on their race. Gould’s analysis took no other factors into consideration, but like Cartwright found that “Whites” had higher lung capacities than “Full Blacks.”[iii] Fast forward to the late 20th century, to an asbestos lawsuit in Baltimore in 1999. The attorney for the defendant, an asbestos manufacturer, requested that “race correction” be applied before compensation be sought for a Black plaintiff. Medical textbooks perpetuated this falsehood, citing an innate difference in lung capacity and function based on race.

Considering these examples of institutionalized racism in the history of healthcare it becomes clear that AI systems, i.e., algorithms are built on large quantities of data which inherently may contain less overt institutionalized racism and could therefore perpetuate racial bias. “Objective” computational methods can easily provide a false sense of partiality for practice standards.[iv]

An article by Obermeyer et al. in Science titled “Dissecting racial bias in an algorithm used to manage the health of populations,” found that the algorithm for a widely used risk assessment tool was significantly racially biased. This commercial algorithm, though unnamed in the study, is used by many health systems throughout the United States. It identifies patients who have complex medical needs for “high-risk care management” programs. The ultimate goal is to allocate resources in a cost-effective manner. This algorithm is often deemed by health systems as “the cornerstone of population health management efforts…improving outcomes and satisfaction while reducing costs.” Many health systems rely on it to identify individuals who will benefit most from additional medical resources.[v] The researchers found that at a certain risk score, Black patients are considerably sicker than White patients, and when correcting for this disparity, the number of Black patients identified as needing extra care would increase from 17.7 percent to 46.5 percent. Obermeyer and colleagues examined the inputs and outputs of the algorithm as well as the outcome data. They investigated how the algorithm was formed to ultimately reveal the mechanism that caused the racial disparity embedded in this risk assessment tool.

The central flaw perpetuating racial disparities in this algorithm was found to be what is known in data science as the “problem formulation” (which in this case, equates healthcare costs to health care needs).  Black patients spend less on healthcare due to structural barriers, discrimination, and distrust, all of which are well-documented issues. Therefore, the use of healthcare cost to predict healthcare needs in the algorithm perpetuates racial disparity. Relying on the algorithm to determine necessary care leads to Black patients receiving fewer services. The algorithm has a circular component: Black people forgoing necessary care in the past is used to predict they need less care in the present or future. Obermeyer et al. used different predictor variables which diminished the disparity. The unnamed manufacturer said it would consider this finding in future rounds of algorithm development. This research shows that the use of AI presents serious ethical breaches. The research was made possible because researchers had access to how the algorithm was formed. The algorithm “black box” relies on quality representative data and accurate predictor variables.[vi]            

It is imperative that the quality of data is representative of the actual population and takes into account the multitude of factors that contribute to, and significantly impact, health outcomes. While we will need counter-bias algorithm tests to correct for subtle systemic discrimination, we also need greater emphasis and training on institutionalized racism in medicine. AI as a tool could deepen and perpetuate racial bias if racial historical contexts are not considered in the construction and testing of the algorithms.[vii] Data sets must include minority and marginalized populations with attention to historical biases as well. Lastly, a concerted effort must be made to increase the diversity of AI engineers so the tools being developed do not merely represent the backgrounds and needs of select groups. This new frontier of AI in health care demands even greater regulatory oversight and concerted efforts to address the issues of implicit biases.

[i] Garg, Shikha, Lindsay Kim, Michael Whitaker, Alissa O’Halloran, Charisse Cummings, Rachel Holstein, Mila Prill, et al. Hospitalization Rates and Characteristics of Patients Hospitalized with Laboratory-Confirmed Coronavirus Disease 2019 — Covid-Net, 14 States, March 1–30, 2020. Center For Disease Control and Prevention (ePublished: April 8 2020).

[ii] Eligon, John, Audra  Burch, Dionne  Searcey, and Richard Oppel. "Black Americans Face Alarming Rates of

Coronavirus Infections in Some States." New York Times, April 7 2020. https://www.nytimes.com/2020/04/07/us/coronavirus-race.html

[iii] Lujan, Heidi L., and Stephen E. DiCarlo. "Science Reflects History as Society Influences Science: Brief History of “Race,” “Race Correction,” and the Spirometer." Advances in Physiology Education 42, no. 2 (2018): 163-65. https://doi.org/10.1152/advan.00196.2017

https://journals.physiology.org/doi/abs/10.1152/advan.00196.2017 

[iv] Yu, Kun-Hsing, and Isaac Kohane. "Framing the Challenges of Artificial Intelligence in Medicine." [In eng]. BMJ Qual Saf 28, no. 3 (Mar 2019): 238-41. https://doi.org/10.1136/bmjqs-2018-008551 

[v] Obermeyer, Ziad, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations." Science 366, no. 6464 (2019): 447-53. https://doi.org/10.1126/science.aax2342. https://science.sciencemag.org/content/sci/366/6464/447.full.pdf

[vi] Obermeyer, Ziad, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations." Science 366, no. 6464 (2019): 447-53. https://doi.org/10.1126/science.aax2342. https://science.sciencemag.org/content/sci/366/6464/447.full.pdf 

[vii] Khullar, Dhruv. "A.I. Could Worsen Health Disparities." New York Times, Jauary 3 2019. https://www.nytimes.com/2019/01/31/opinion/ai-bias-healthcare.html