We partnered with Cardiff University’s Hartree Hub to find an AI solution for delivering post-play debriefings for serious games.
Serious games have the potential to facilitate learning and encourage critical reflection. They do this in a number of ways, such as using game mechanics to deliver information in a playful and structured manner and by positioning players as active, decision-making agents in game narratives. However, it is difficult to assess what a player has learned by playing a serious game or the extent to which the game has inspired them to challenge their perceptions and behaviours (if at all).
In our experience of developing serious games, we’ve found that one of the more reliable ways to evaluate a serious game’s capacity to stimulate learning and critical thinking is a post-play debriefing. Yet, conventional debriefs, in which a human facilitates a conversation with one or more players, are not always feasible. There are many practical reasons, including cost, researcher and player availability, sustainability, and scalability issues. These barriers not only limit the scope of data collection and evaluation in serious games but also increase the likelihood that players walk away from their experience having not reflected on game content at all.
To find an innovative solution to this problem, we worked with computer scientists at Cardiff University’s Hartree Hub to explore how Large Language Models (LLMs) can be utilised to create an automated debriefing tool for serious games.
What are LLMS?
At their core, LLMs are a form of Generative AI - that is, computer systems that can create new content from vast amounts of data. GPT-4o, the technology behind OpenAI’s ChatGPT, is one example of a large language model. LLMs generate content by starting from some text, e.g., a sentence or paragraph, and then making predictions about what words should come next, each time choosing the most likely word based on the context. Consider this example and ask yourself what word comes next: ‘The sky is __.’ Literally, any word could come next, but we may agree that ‘blue’ is more likely than ‘yellow’ or ‘pancake.’
How it works
The idea is inspired by Reinforcement Learning, a machine learning algorithm that works by rating an AI every time it makes a decision. The AI is configured to maximise its rating and thus gravitates towards decisions that give it high ratings, in turn ‘learning’ how to make good decisions. In our scenario, the LLM is learning to make good decisions about what questions to ask a player based on the ratings.
By leveraging LLMs, we can simulate a facilitator who interacts with players in a post-game debrief scenario. After playing, the game prompts the player to a ‘debrief’ opportunity to discuss their thoughts and feelings about the game. The AI asks open-ended questions to stimulate reflection on the player experience. For example, if we know the game drives players through a series of difficult decisions, the system may question the player’s choices. This is the key phase. Based on the player’s responses, the AI can ask follow-up questions to delve deeper into specific topics, experiences, or choices. If a player mentions a specific choice, the AI might ask, ‘What happened after you took this decision?’
In addition to providing players with an opportunity to reflect on their game experience, the system may also collect data to help inform designers how the game may better convey its ‘serious’ message. This includes identifying issues related to gameplay (e.g., functionality, balance), UI (e.g., readability, accessibility, control), and performance, alongside impressions on graphics, sound, and overall enjoyment of the game. All interactions between player and the system could be recorded and analysed, enabling researchers to gather large amounts of player feedback at scale.
Enhancing impact
Conventional debriefings led by human facilitators are often not feasible, especially for smaller game companies and their partners (e.g., museums, schools, hospitals) due to limited resources. An automated AI solution makes the debriefing process accessible and scalable, enhancing the capacity to stimulate critical reflection and with it the impact of a serious game. AI facilitators can handle multiple players simultaneously and work around the clock. Furthermore, the dynamic nature of AI-driven interviews allows for a deep exploration of player experiences. For example, an AI can be tuned to uncover and analyse a player’s emotional highs and lows across the game, thereby providing a more holistic view of how the game has impacted them.
Next steps
At Echo Games CIC, we’re constantly challenging ourselves to improve not just the games we make, but also how we effectively convey their ‘serious’ message. Partnering with Cardiff University has enabled us to further this mission by helping us create new tools for conducting rigorous games research and enhancing the effectiveness of serious games. We’re excited to continue working with the Hartree Hub to enhance the versatility and accuracy of an automated debrief, applying it to a range of games, and testing prototypes with a range of player demographics. More to follow!
