At the end of October 2024, I had the pleasure of speaking on a keynote panel session as part of RExChange 2024 on what AI means for knowledge in the religion and worldviews classroom. I was speaking alongside Professor Beth Singler – who did an excellent job of providing a sweeping overview of almost everything “AI” could mean for the study of religion, with some fascinating examples – and Dr Michael Burdett – who provided a thought-provoking discussion of various ethical and philosophical issues presented by AI for the teaching of “religion”, broadly conceived. My contribution was sandwiched between these two and, given that much of my thinking now is taken up with the OU student experience, and potential uses and abuses of generative AI, I decided to engage with ChatGPT head on, from a critical religious studies perspective. You can view the full session and ensuing discussion here.
Before going much further, I should state that by “critical religious studies perspective” I mean one like we adopt at the Open University which acknowledges, to quote Jonathan Jong, that
“there is no such thing that answers to the name ‘religion’”, but only phenomena that “we habitually label religious” for historically contingent reasons (Jong 2015, 20).
In his A Critical Introduction to the Study of Religion, Craig Martin (2017, 156) argues that rather than asking questions like, for example, “Is she Catholic?” we should ask:
- Who is identified by
- Whom as
- What, and
- With what social effects?
These are the sorts of questions that animated my recent engagement with ChatGPT (currently in its fourth iteration – “GPT-4”), which is a large language model that can do everything from “answering” questions and “composing” songs, to summarizing texts and attempting to answer essay questions (all to varying degrees of success).
Asking GPT-4 about religion: the good
First off, it should be noted that when prompted with a variety of questions related to the category of “religion”, much of what GPT-4 comes up with is pretty good. It is, after all, basically doing your Googling for you and producing a summary of material available to it.
[Side note: its results should thus be treated with the same level of care and scepticism with which good scholarship would treat any web page].
For example, when I asked it “what is religion?”, it produced a defensible list (with definitions) of what the concept “typically involves” – beliefs, practices, moral and ethical guidelines, community, sacred texts and traditions, and spiritual experience – and concluded that the “specifics of what constitutes a religion can vary greatly between different cultures and belief systems, and the boundaries of what is considered a religion can be fluid”. Sure, there are issues with this, but as a general introduction, I was quite surprised at the level of nuance. I was similarly pleased with responses to prompts such as “What do Muslims believe?”, “Is religion a force for good?”, or even “design a teaching activity for 12-year-olds on the nature of religion”.
Asking GPT-4 about religion: the bad
On the other hand, there are times where the responses are woefully bad. For example, I asked it about my own work with the prompt “What is Christopher R. Cotter’s perspective on religion?” It did get some things right – that I adopt a critical interdisciplinary approach “drawing from sociology [yes], religious studies [yes], and cultural studies [kind of]”, and that I am interested in “a deeper understanding of how religion functions in contemporary society and how it intersects with various social and cultural phenomena” [yes, but vague].
It also informed me that I am interested in how “new religious movements interact with and differ from traditional religions” [I mean, I am, but not in much depth] and attributed the book “Understanding Religion: Theories and Methods” to myself and someone called Matt James [I have not written this book, and do not know a Matt James, although the title is a partial copy of Paul Hedges’ recent work Understanding Religion: Theories and Methods for Studying Religiously Diverse Societies]. This is an example of what is known as an AI hallucination.
Asking GPT-4 about religion: the ugly
Finally, some of what GPT-4 produces is worryingly problematic from a critical perspective. I’ll share three examples.
Thinking about common dietary prohibitions, I prompted GPT-4 with “Do Muslims eat pork?” and “Do Jews eat pork?” Again, the answers provided were not terrible, acknowledging the history and precedent of this dietary prohibition in these traditions, as well as reflecting on its observance and cultural impact. However, in response to the “Muslims” prompt, I was immediately greeted with the definitive statement “No, Muslims do not eat pork”, with GPT-4 concluding that “Muslims worldwide adhere to this prohibition.” When “Jews” was substituted into the prompt, the opening gambit was “No, observant Jews do not eat pork”, with the conclusion being that the prohibition is “upheld by observant Jews.” This prompted some questions:
- Why is one group treated as a monolith (i.e., that they all do this), while the other is allowed some nuance via the qualifier “observant”?
- If someone identifies as Muslim but does eat pork, are they then excluded from the category of Muslim?
- What does it mean to be an “observant Jew”, and can one not be “observant” whilst also eating pork?
- Who decides on these boundary markers, and what will their social effects be?
- Why is GPT-4 being so definitive, when we know that there are numerous, sincere, self-identifying Muslims and Jews who do consume pork, and who do not consider this to be a problem?
Turning to another prompt, when I asked “Do Muslims drink alcohol?”, I was greeted with a definitive “No, Muslims do not drink alcohol” and, after some useful historical context, the conclusion was that “practicing Muslims abstain from drinking alcohol as part of their adherence to Islamic teachings.” This time GPT-4 is willing to acknowledge some Muslim diversity with the addition of “practicing” but, again, the implication is that Muslims who do drink alcohol are not practicing – and therefore, presumably, less authentic? – even though there are many sincere “practicing Muslims” who do consume alcohol. Yes, the consumption of alcohol would be condemned by many Muslim authorities, but someone consulting GPT-4 would come away with the conclusion that these voices were the only correct ones, and that those who might oppose them – if they exist at all – are lesser in some way.
Finally, when prompted with “Do Catholics have premarital sex?” the immediate response has a different inflection: “The official teaching of the Catholic Church is that premarital sex is not permissible.” After discussing some of the reasoning behind this stance, GPT-4 then acknowledged that “it’s important to recognize that individual Catholics may have varying levels of adherence to this teaching. In practice, some Catholics may engage in premarital sex, but this is considered contrary to Church teachings…”. Here, we see diversity acknowledged, and no exclusion taking place: officially they aren’t supposed to, but many do. Why is there no qualifier “practicing” or “observant” added here, when it was for Muslims and Jews? Why is the acknowledgement of “lived” digression from “official” teachings made so effortlessly here, and not in those other cases?
The simple fact is that GPT-4’s responses are dictated by the material it has been “trained” with, the presumptions of the team behind its coding and production, and the prompts provided by users. And thus, the biases, stereotypes and emphases that dominate in each of these arenas – explicitly or implicitly – will literally be written into the responses GPT-4 produces.
Above, I quoted Craig Martin’s critical questions:
- Who is identified by
- Whom as
- What, and
- With what social effects? (2017, 156)
Using the example of Muslims and alcohol consumption, this results in the following:
- Muslims who drink alcohol are identified by the team behind Chat-GPT as not being “practicing Muslims”, resulting in their exclusion from the dominant model of “proper” Islam, and the potential perpetuation of stereotypes, disapproval, and reprobation from within and outside the “Muslim community”.