Blog

Festival of Learning Insights

Seoul, South Korea, June 26 – July 3, 2026

By Terry Junsoo Park

Practical takeaways:

  • Public datasets – we should familiarize with them and make use of them to test our ideas before applying them in deployment, and for comparisons.
  • Hardware – we can look at the user interface, or the “embodiment” aspect of our tools more, as we naturally understand things better when we see it in action with our own eyes, rather than hearing or reading about it.
  • There is still a gap in using data effectively for personalization while preserving privacy. Some works seem to ignore it entirely and focus on the learning effects, but in the real world, privacy is a real concern and there should be a real considerations of the trade-offs between personalization and privacy.
  • Most of the works with results on learning effects seemed to have access to a lot of structured data, something which we are not working with.
  • Due to the hard work at DILab, it’s likely that we will soon have structured data and outcome labels, but the current situation which is a constrained resource setting is actually ideal for discovering novel solutions, while it might be frustrating to get anything done.
  • Embrace ambiguity, uncertainty, and challenge, because if it’s easy, it’s no longer interesting.

Day 1: HAI-Agency Workshop on Orchestrating Human and AI Agency for Proactive and Reflective Learning

Dr. Lixiang Yan, the keynote speaker from Tsinghua University, introduced the concept of agentivism and shared empirical results from their CoLearn program. It was interesting to see how a multi-agent system was deployed in their classroom environment, and how they measured the learning effects from a pre- post-study. This is insightful for our own setting where learning is collaborative, multi-agent, and confounds are a real challenge. It also made me realize that agent personas could be a meaningful design knob, or experimental condition to be studied independently. I would be interested to see studies on more structured coordination in multi-agent systems, to see the isolated effects of different sequences or compositions of human-AI agents on student learning.

Notable presentations

Explainable AI Mediation through Knowledge Graph Design for Psychoeducational Intervention Co-Creation

Authors: Triet Bui, Jue Xie, Jia Rong

This was a great presentation which illustrated how to construct different knowledge graphs using tools such as Neo4j to address data curation bottlenecks and improve explainability, with the parent of students as the end user in mind. The authors compared the effects of using chunk-based vs proposition-based graphs in their pipeline and evaluated them on characteristics of the representation itself vs qualitative outcomes on users in a workshop. They found that chunk-based representations were better for factual referencing and traceability, while proposition-based were better for generative flexibility and natural dialogues. This gives insight into the trade-offs of different graph representation design choices, as well as informs our future work in using knowledge graphs for steering LLMs and other use cases.

Key Discussions

After my presentation, a researcher from Seoul University asked me why I described our work on the Jill Watson dashboard as co-agency. The underlying question was how was AI actually supporting the teacher’s agency, rather than automating their work.

I view Jill Watson as having the agency to support students of online classrooms in real-time, 1-on-1 conversations, especially handling common questions which does not require much critical thinking on either the student or the teacher’s end. The Jill Watson dashboard supports the teacher’s agency by escalating student needs that Jill Watson can’t resolve along with the context of the student’s interaction, like what they tried up to that point, so that teachers can focus on personalized instructional support for these cases that need the teacher’s expertise. Without a teacher-facing learner model like the Jill Watson dashboard, the teacher would not have this conversational history in online classrooms, and students would not have their needs addressed by the teacher if Jill Watson could not provide a solution.

Day 2: CSEDM Workshop @ EDM’26

This workshop was about educational data mining in CS. It was more focused on technical aspects and system details. I found it to be a great learning opportunity as I would rate myself stronger on the learning science and theory side of things rather than the technical. One recurring pattern was that while the technical contributions were impressive, most of the authors had plenty of student data, all the way down to code-level attempts on CS assignments in introductory programming courses. This is a bit different from our case when working with free-response text in higher-level courses, and at times, anonymous data. It did give clarity on what we would do, if we either had that data, or could curate the data we had into a form we could plug-and-chug.

Notable presentations

SHARP: Socratic Hinting for AI-Reinforced Programming in Competitive Programming Education. 

Authors: Yuchen Wang, Chiraag Singh Anand and Chee Wei Tan

Code: https://github.com/Snail664/coding-tutor

This was a very relevant work as they demonstrated how to apply an LLM-based recommender system-like pattern in a classroom setting. The 3 components were: Socratic hint generation, hint scheduling, and student progress tracker in a continuous feedback loop. The most valuable component was the milestone division using language model embeddings, and student progress inference using semantic similarity, something very similar to our own work on detecting student needs and knowledge gaps. Another takeaway was what intervention the system should generate, which was in this case was Socratic hints.

Operationalizing Misconception Catalogs for LLM Distractor Generation in Introductory Python

Authors: Bryan Hollingsworth, Peter Brusilovsky, Jaromir Savelka

Automated Recommendation of Programming Learning Content Using Pattern-based Knowledge Components

Authors: Muntasir Hoq, Griffin Pitts, Zhangqi Duan, Arun Balajiee Lekshmi Narayanan, Mohammad Hassany, Andrew Lan, Peter Brusilovsky and Bita Akram

Ontology-Guided Knowledge Component Extraction for Programming Exercises

Authors: Rully Agus Hendrawan, Shravya Suresh, Rafaella Sampaio de Alencar, Alice Micheli, Peter Brusilovsky and Sergey Sosnovsky

The takeaway from the 3 papers above was how AIED systems could intervene in response to student needs, and how we can measure learning effects. Knowledge components seems to be a useful intermediary representation, while distractors to identify student misconceptions was an interesting use case that, frankly, I heard about for the first time.

Key Discussions

Upon presenting our work on the knowledge gap detection using pre-requisite graphs, I received plenty of questions and feedback.

Some takeaways:

  • Future work on automating pre-requisite graph from course textbook and syllabus can be done with tools such as neo4j, we actually do have a “ground truth”, which are syllabi that teachers produce based on the textbook.
  • How do we identify the “depth” of the student’s knowledge gap? As we are now in a graph representation world, an idea is that we can just apply graph algorithms and walk up the tree to identify root causes.

I had a fruitful discussion with another researcher who was working on a similar problem, but perhaps more difficult, for 4 years. They were trying to create concept maps from teacher artifacts such as powerpoint presentations, and they mentioned that this is far more complicated than when we were provided a textbook with a structured table of contents, etc. As presentation slides can be a mix of text, images, and even different structures like tables, formulas, and teachers may not design a course in a linear sequence. While we brainstormed some ideas, this remains future work, as I face a similar problem in analyzing student assignment submissions in KBAI. Hopefully I can solve it in less than 4 years!

Day 3: AIED @ Festival of Learning

I presented our work on evaluating learner representations prior to deployment. I was happy that our work seemed relatively novel compared to what I had seen, as our contribution was before learning effects and focused on studying representations before modelling and direct outcomes. Some sharp questions I had received were how we constructed the learner representations, defined engagement, and how our metric distinctiveness could be applied to other courses such as math.

The core pedagogical features in our representation were validated with classroom data from the same course but with a prior cohort, and our learner representations are an aggregate of models specializing in 1 specific pedagogical features, such as logistic regression, trained to predict engagement risk. Engagement was defined as later participation (frequency of their questions), and yes, distinctiveness can be applied to other courses as it is derived from pairwise comparisons of their interactions with Jill Watson, with no prior assumptions on distance, but “scales” based on the student’s engagement relative to their peers. I guess the next step really is to see if that hypothesis is true, and test it on another class!

Notable presentations

From Feasibility to Generalization in the Field: Using an MR-Based Teacher Orchestration Tool in AI-Supported K-12 Classrooms

Authors: Qiao Jin, Yang Zhan, Will Morgus, Kyle Price, Michael Sandbothe, Jonathan Sewall, Octav Popescu, Susan Berman, Stephen E Fancsali, Steve Ritter, Ken Holstein, Vincent Aleven

This was an exciting presentation from CMU, with co-authors Ken Holstein and Vincent Aleven, who I have cited in the past. Basically, they have an intelligent teaching system called Mathia, and they have connected it with mixed reality glasses (X-Real), and it indicates in real-time, in augmented reality, student status, such as doing well, unproductively struggling, struggling, and idling. It demonstrated how the systems we develop can be extended to hardware so that teachers can use them in their day-to-day work. I can see how with a little engineering, the tools we build could live in the classroom and beyond in the products that we are already familiar with. Seeing the demo made me intuitively grasp how close we are to that point.

Beyond One-Size-Fits-All: Personalizing Computational Proxies of the Zone of Proximal Development Through Help-Seeking Behaviors

Authors: Kaimao Sheng and Irene-Angelica Chounta

I loved this presentation because it demonstrated an alternate trajectory I could have had at DILab. In my first semester, I was obsessed with Vygotsky’s Zone of Proximal Development, but eventually pivoted my research focus. This demonstration was exactly the implementation of ZPD, my initial obsession. I.e. the authors used a public dataset (DataShop) and modeled it in such a way that the ZPD could be visualized to teachers (and we could use those boundaries to adapt instructional interventions). The dataset comes with it learning outcomes such as hint-correct, hint-hint, and hint-incorrect sequences based on how students behaved in response to 1st hint opportunities in quiz attempts. This gives us a way to study the effect that hints have on learners and when to best provide hints. However, I see the limitation which is that it depends on these labels to be provided in advance. In our setting, had I continued on this path, I probably would have ran into the wall of having no labels for the real classroom data we have, and would have had to use a public dataset. Coming back to Vygotsky though, I’m interested in seeing how we can use it for studying intervention pacing effectiveness using the tools we’ve made such as distinctiveness and our twin network-based classroom simulator pipeline.

Key Discussions

I had the chance to discuss with Vincent Aleven later in the conference and I did ask about the Mixed Reality work. Sorry to be a downer, but I asked, what if the student was shown as idling, when they were actually just thinking? Vincent was aware of these limitations, and said that they did find that students who spent more time reading and understanding the problem actually did better, so the accuracy (is that the right word here?) remains future work.

Day 4: L@S @ Festival of Learning

Ashok’s presentation was very well received. I was able to see him at work and in-person for the first time. I could see why he has had such an impact on the field. A great and inspiring experience that I’d recommend to anyone. He had the audience’s complete attention, and people were lining up to talk to him afterwards. What impresses me is how long he has stuck with this, long before everyone knew about ChatGPT.

Notable presentations

Developing Models of Procedural Skills using an AI-assisted Text-to-Model Approach

Authors: Rahul Dass, Shubham Puri, Arpit Khandelwal, Xiao Jin, Ashok Goel

As DILab works on quite a few projects, quite frankly, I didn’t know what TMK was until Rahul’s explanation. There’s a lot of details on how it is actually produced, but my takeaway is that it is an abstraction for turning data into a representation (for procedural skills) which can be used more effectively in downstream modelling. On the job, I use coding agents daily, and the way I view TMK now is that it’s an “agent skill” for DILab’s educational AI tools. An agent skill is really just a prompt with some embedded knowledge representation which is reusable, so I think this is the right analogy. I wonder if TMK representation can be (or if others in the lab are already) encoded directly as a graph using Neo4j and similar tools. I think Text-to-Model is an interesting area to explore, or even Text-to-Representation (representation learning). Future work for sure.

Multistage Modeling from Application Signals to Downstream Success: Predicting Admission, Matriculation, and Retention

Author: David Joyner, Alex Duncan

David Joyner’s presentation was the first presentation I saw at the conference that showed the findings from a large sample size. They took 50000 records of OMSCS of applicant demographics as well as text features from their applications and investigated if program success could be predicted at admission time, and with 1st term progress. The takeaway is that success can’t be predicted from the admission, but 1st term momentum makes long-term success in the program explainable. In our work, we do something similar but within a classroom, but I wonder if we can investigate more detailed reports or recommendations at key decision times like at each quarter of the semester.

I asked David about how trajectories differed within specializations, as I guessed that specializations which included Graduate Algorithms would look much different between matriculation and graduation than the specializations that did not. Their study didn’t include graduation data, but he mentioned that they noticed interesting shifts in the population when the interactive computing specialization was renamed to the AI specialization, where those that might have performed poorly in a GA-based specialization shifted to the AI specialization. My view on distribution statistics is that measures might not fully explain an individual’s experience, while group behavior might be more predictable than an individual might expect within their individual experience.

Day 5: Sponsor Panel @ Festival of Learning

On Wednesday, July 1st, I attended the Sponsor Panel and had discussions with Google researchers.

Notable presentations

I attended the sponsor talks from Google Deepmind and Google Brain, where they presented their work on Gemini’s guided learning and study notebooks. The way I saw it was that they were solving a very similar problem to Jill Watson and the Jill Watson Dashboard. I was a bit surprised that Google Brain was using the same terminology (meso-learning) from AI-ALOE. I’ve actually used Google’s NotebookLM for studying and I think it’s pretty impressive, especially in its retrieval, although I think its quiz generation is lacking. I was able to have a discussion with the presenters, and here were the key questions:

Q: In their work, what is currently missing that they really want someone to solve?

A: Representations of pedagogical intent.

Q: In their work in meso-learning, what do they find to be the most important thing when doing memory design in AI systems?

A: Starting from the teacher’s theory of change, from there you can design a memory architecture suited for that theory of change.

Key Discussions

I had a good discussion with David Nahodyl, another student at Georgia Tech who works with Dr. Thad Starner. It was interesting to hear how they were using their NOSI IDE in the AI class, and we discussed what they looked for in student interactions in the IDE. We thought breakpoints would be an interesting thing to measure, and also thought it would be really cool to put one of DI Lab’s AI agents inside NOSI, kind of like a Copilot in IDEs nowadays.

We also discussed CS education. I think a challenge currently is that CS undergraduate education does not train students to become a software engineer or get a job in industry. I don’t know if it ever did, and of course, not everyone studies CS to become an engineer. However, if you are going to school in CS to be a software engineer, and schools don’t let you use AI, you will be competing against new grads who do know how to use AI. Industry doesn’t really care if you understand CS theories deeply, they optimize for and reward efficiency. Using AI is highly efficient most of the time. The times it’s not is when AI can’t solve a problem. Here’s a conundrum… If you want to be a software engineer in today’s world, you need to not only be able to use AI for tasks it can do well, but you need to be better than AI, so you can solve the problems it can’t. If the goal is software engineering, how can you start from not knowing how to code, to becoming better at coding than AI in 4 years?

Of course, there’s more to software engineering, and new grads can become good at writing and being able to explain simple, clean code that gets merged, but my questions are now:

  • Is the skill of AI literacy/utilization a better skill to optimize for than coding from scratch?
  • Aren’t they both forms of problem solving, but at different levels of abstraction?
  • If schools don’t teach students how to use AI to solve problems and ban them outright, how much are they setting the students back, or is that actually better for student success after graduation?

Day 6: AIED @ Festival of Learning

I attended Vincent Aleven’s keynote at AIED, as well as the L@S conference for Rahul and David Joyner’s presentations.

Notable presentations

I enjoyed Vincent Aleven’s keynote presentation on human-centered AIED frameworks. Something that didn’t even occur to me before as we work in adult learning, is that parents are a key stakeholder in the student’s learning ecosystem. If we consider a workplace setting, it makes sense, the “parent” might be your manager. In higher education, this role is not so clear. Maybe it’s a more philosophical question, but does this role exist for students in higher ed? Are adult learners performing dual roles in this case, and by improving the student’s caregiver in this case, are we actually improving the part of them that will determine their future success?

Key Discussions

There was a part of the presentation which explained how there was a drop in grades after the rise in AI (I can’t remember the details of where this study took place). It made me wonder if it was AI that was at fault, or if it was that teachers were just grading more harshly. Grades are after all something that the teacher assigns, even if it is rubric based. A question I have is if schools are evaluating the right skills at all for an AI-augmented workforce, and if schools have ever evaluated the right skills for long-term student success.

Day 7: AIED @ Festival of Learning

I had the chance to check out another conference at the same venue on the last day, which was more startup focused. It was fun to see how others were using AI in products that addressed peoples’ problems outside of education, even those that were not inconveniences, but ones you might still pay to be addressed. It reminded me that it might be the startups or big companies that actually bring the work that we do to the everyday person, but also, it’s important to not always take things too seriously, and also work on some things purely for your own enjoyment, meaningless as they might be. You never know where you might get inspiration.

Papers to Read

The top 8 papers I want to read next from the conference:

Time as a geometric property of the latent space: An empirical study of Time-and-Order RoPE in Knowledge Tracing

Authors: Willian Costa, Filipe Dwan Pereira, Gabriel Alves, André Nascimento, Luiz Rodrigues, Mario A. P. Santos, Tyagi M. Lima, Rafael Ferreira Mello

Federated Code-Aware Knowledge Tracing for Privacy-Preserving Student Performance Forecasting

Authors: Anurata Prabha Hridi, Muntasir Hoq, Seyyedali Hosseinalipour, Rajeev Sahay, Bita Akram

Measuring Creativity at Scale via Multimodal Large Language Models

Authors: Armanda Lewis and Xavier Ochoa

Misconception Diagnosis From Student-Tutor Dialogue: Generate, Retrieve, Rerank

Authors: Joshua Mitton, Prarthana Bhattacharyya, Digory Smith, S. Thomas Christie, Ralph Abboud and Simon Woodhead

Estimating Learners’ Skill Acquisition Without Temporal Information

Authors: Ryosuke Nagai, Kyohei Atarashi, Koh Takeuchi, Jill-Jênn Vie, Hisashi Kashima

Fair When Sparse, Accurate When Rich: RL-Gated Conversational Fusion for Temporally Equitable Knowledge Tracing

Authors: Fan Zhang, Rui Guo, Hai Li

Spatiotemporal Link Formation Prediction in Social Learning Networks Using Graph Neural Networks

Authors: Ali Mohammadiasl, Bita Akram, Seyyedali Hosseinalipour, Rajeev Sahay

Permutation-Based Significance Testing for Temporal Sequential Patterns in Learning Event Logs

Authors: Tianwei Peng, Jinwang Song, Sijing Yu, Sijie Zhang and Yingbin Zhang

Personal Reflection

One of my biggest takeaways from the conference was that presenting the papers was only the beginning. The real work came after: asking questions, discussing ideas, learning from other researchers, receiving feedback, and building new connections. I feel like I finally got a taste of what it means to do research. It’s also worth acknowledging that I wouldn’t have been at this conference without Georgia Tech’s OMSCS program.

Many thanks to Dr. Ashok Goel for his continued guidance and support throughout this journey. Acknowledgements to the team which I represented: Htet Phyo Wai, Youssef Medhat, Ploy Thajchayapong.


Papers Presented


Park, J., Medhat, Y., Wai, H.P., Thajchayapong, P., Goel, A.K. (2027). Evaluating Learner Representations for Differentiation Prior to Instructional Outcomes. In: Blanchard, E.G., Chen, G., Chi, M., Isotani, S. (eds) Artificial Intelligence in Education. AIED 2026. Lecture Notes in Computer Science(), vol 16585. Springer, Cham. https://doi.org/10.1007/978-3-032-29770-9_4.

Medhat, Y., Park, J., Thajchayapong, P. and Goel, A.K., 2026. Detecting Knowledge Gaps from Conversational AI Interactions Using Curriculum Prerequisite Graphs. arXiv preprint arXiv:2606.10736.


Park, J., Medhat, Y., Wai, H.P., Thajchayapong, P. and Goel, A.K., 2026. Surfacing Isolated Learners with Outcome-Independent Mediation of Feedback between Teachers and Students Using AI. arXiv preprint arXiv:260

AAAI 2025 Spring Symposium Report

Research Scientist, Ploy Thajchayapong, attended the AAAI 2025 Spring Symposium in San Francisco, California. Below are her brief observations as well as the talks she found most interesting.

Observations:

  • The symposium had a relatively small number of participants
  • Much of the work presented was still in the early stages, with limited results
  • A noticeable trend was the use of gaming environments as testbeds for studying human-AI collaboration
  • None of the projects focused on learning environments or utilized student data to explore collaboration in that domain.

Talks:

Johnson and Bradshaw – Toward a Taxonomy of Collaboration
This framework introduces a taxonomy to classify types of human-agent collaboration based on roles, authority, and decision-making structures. It emphasizes adjustable autonomy, where control can shift dynamically between humans and AI depending on context, trust, and task complexity.

Huang – Toward a Generalized Model for Human-AI Collaboration
Huang proposes a generalized model that focuses on key collaborative components: goals, capabilities, models (of self and partner), and communication. This model aims to support predictability, directability, and adaptability in AI agents

Verma et al. – Developing Shared Mental Models
Verma and colleagues focused on the development of shared mental models between humans and AI to support effective coordination. The talk highlighted how mutual understanding of goals, capabilities, and expectations is key to reducing miscommunication and increasing trust.

Chen – Hybrid Machine- and Human-Centric Approaches
Chen’s framework proposed a hybrid paradigm that balances human strengths (e.g., contextual judgment, creativity) with machine capabilities (e.g., speed, consistency, scalability). The talk outlined design strategies where AI supports rather than replaces human judgment, advocating for human-centered AI that is adaptable, transparent, and accountable. Emphasis was placed on creating systems that can co-evolve with human collaborators.

Favier et al. – Leveraging LLMs for Decision Making
This talk examined how large language models (LLMs) can be integrated into decision-making pipelines, particularly in complex or ambiguous scenarios. The authors emphasized the potential of LLMs to synthesize information, generate options, and offer justifications, acting as co-pilots for human decision-makers.

Qin and Sajda – Teaming with AI
This talk provided a broader perspective on human-AI teaming in decision-rich environments, such as healthcare or high-stakes operational settings. Qin and Sajda emphasized shared situational awareness, cognitive load balancing, and adaptive task allocation as keys to successful collaboration. They discussed empirical studies showing that optimal teaming depends not just on AI competence, but on timing, communication, and trust calibration throughout the decision-making cycle.

Morey – Ask Not What AI Can Do For You
Morey offered a provocative and reflective critique of dominant narratives in AI. Rather than asking what AI can do for humans, the talk reframed the question to explore what humans want AI to do, and at what cost—including ethical, social, and cognitive impacts. Morey emphasized the need for intentional design that prioritizes human agency, values, and long-term well-being over sheer technical capability or convenience.

Zamprogno – Human-AI Collaboration for Discovery
This talk explored how humans and AI can co-create new knowledge, especially in scientific and exploratory domains. Zamprogno emphasized the power of AI to augment curiosity-driven inquiry by uncovering patterns, generating hypotheses, or simulating scenarios. Successful discovery requires interactive exploration, where AI remains flexible, interpretable, and responsive to evolving human goals.

Seo et al. – AI-Assisted Coordination of Human Teams
This talk focused on the use of AI to facilitate collaboration among human team members, acting as a meta-coordinator. Their system monitored team dynamics, detected communication gaps, and suggested interventions to improve task flow and role clarity. The goal was not to replace human leadership but to augment team performance through unobtrusive, adaptive support.

Research Scientist, Jeanette Luu, Attends Tech AI Fest

DILab Research Scientist, Jeanette Luu, attended the three day Tech AI Fest at Georgia Institute of Technology, March 26–28, 2025. She shares her comments around the Fest’s forums and presentations.

Tech AI Fest Day 0 

OpenAI Workshop OpenAI Academy

  • The new features include deep search, reasoning, visual LLM where you can upload video. The workshop was informative and showy.
  • Sora short films screening. Panel discussion with film makers, moderated by Dr. Mark Reidl.
  • Those short films are impressive and have their own documentary flare. Although the film makers enjoyed working with Sora but they believe that human are still in the loop for film making.

Tech AI Fest Day 1

  • The presentation from Zico Kolter from CMU was interesting. He actually believes that transformer architecture is fine. Pre-trained LLMs can be fine-tuned further so it can be more resilient, less likely to be jailbreak, i.e. produce harmful content such as how to hotwire a car.
  • Presentations from Kinaxis, MindsDB and Juniper focused on the application of GenAI in the industry.

One of the interesting things about Juniper presentation is how does industry evaluate nondeterministic output from GenAI. That reminds me that maybe we need a design guideline for evaluation.

During the panel discussion with the above companies’ presenters, they also mentioned that minimal viable products are not the same as production ready products. MVP must be able to scale and production ready in order for adoption.

Tech AI Fest Day 2

From day 1 and today, one emerging theme is how does AI quantify uncertainty/predict the hidden variables. This goes back to that if we need a design guideline for evaluation on prediction.  The opposite view on this if that if human can make sense of the AI prediction, human does not need to know the decision making of AI. 

One comment is that additional explanation about the AI decision making in natural language actually lead human to incorrect conclusion.

Another theme is that foundational models don’t need to be “know it all” but rather more easily to generalize into specific domain without additional effort, plug-and-play., able to be trained/fine tuned, with less data/less compute power.

For AI adoption, the product/service needs to work the first time consistently, can be scaled readily/easily. When it comes to AI/robotic automation, we need to consider what human is good at and what AI is good at, keeping human in the loop.

A note on the reasoning capability of OpenAI latest model, it has a layer of “reinforcement learning with human feedback and verifiable outputs, such as 1+1=2”. Is that how human reason?

DILab Attends AAAI 2025 Theory of Mind Workshop

During his recent attendance at the AAAI Theory of Mind workshop, DILab member Manuel Preston De Miranda commented on several key speakers and posters that had interesting and insightful knowledge. Below is Manuel’s highlight of the discussions.

Keynote speaker: Rebecca Saxe

Rebecca Saxe discussed that we often view ToM as simply a tool for predicting others’ actions, such as expecting someone to feel upset after a frustrating event. Yet, in her talk she argues that ToM is equally important for planning interventions to change those actions or emotions. In other words, it is not just about understanding what someone might do but also about strategically influencing what they eventually do.

The discussion was particularly compelling when it explained how, in dynamic and unpredictable environments, relying solely on habitual or statistical models of behavior is not enough. Instead, a more abstract and structured causal model of others’ beliefs, desires, and intentions is needed. Dr. Saxe provided concrete examples, such as guiding someone’s actions by altering their underlying beliefs, which is comparable to subtly redirecting attention in order to shift decision-making in high-stakes situations.

She proceeded to list out and answer: How would you know if a system has a ToM? She states to know this you must measure performance on a larger more representative space of tasks including:

     -Continuous, graded, joint inference from sparse data

     -Predicting, explaining, evaluating actions and emotional reactions.

     -Planning interventions to change actions and emotional reactions.

     -Especially in novel situations and large spatial and temporal scales.

She concluded the presentation by asking, “What if an artificial system passed all of the criteria above, using a domain-general representation completely shared with language and reasoning?”

Keynote speaker: Harmen de Weerd

Harmen de Weerd discussed the significance of higher-order Theory of Mind (ToM) reasoning in our capacity to understand that others have beliefs, desires, and intentions, and that they, in turn, can reason about our mental states in dynamic and unpredictable environments. Through agent-based simulations in a negotiation setting known as Colored Trails, the study examines how varying levels of environmental predictability influence the effectiveness of higher-order ToM reasoning.

The findings reveal that in highly predictable environments, agents gain minimal advantage from employing ToM reasoning, as the behavior of others remains relatively constant and foreseeable. However, as the environment becomes more dynamic with observable features that change over time, agents without ToM capabilities struggle to accurately anticipate others’ actions. In these unpredictable settings, agents equipped with higher-order ToM can better interpret and predict behaviors, leading to improved negotiation outcomes. This suggests that the human-specific ability for higher-order ToM reasoning may have evolved to navigate complex and uncertain social interactions effectively.

These insights underscore the adaptive value of higher-order ToM in environments where unpredictability is a constant factor. By enabling individuals to recursively consider the mental states of others, higher-order ToM facilitates more accurate predictions and strategic decisions in social interactions, particularly when dealing with incomplete information or rapidly changing circumstances. This research not only enhances our understanding of human cognitive evolution but also has practical implications for designing artificial intelligence systems capable of sophisticated social reasoning in unpredictable settings.

Keynote speaker: Sheila McIlraith

Sheila McIlraith presented detailed insights into her research on “Proactive Robotic Assistance via Theory of Mind (ToM),” emphasizing the importance of equipping robots with the cognitive ability to understand and reason about human mental states, including beliefs, desires, and intentions. She explained how her team utilized epistemic planning, particularly highlighting their use of the KD45 logic encoding, to allow robots to effectively represent and reason about the beliefs of multiple agents, including potentially incorrect or outdated beliefs. Dr. McIlraith detailed the process of encoding these epistemic states using the PDKB Domain Description Language (PDKBDDL), a specialized variant of PDDL designed specifically for multi-agent epistemic planning scenarios.

Further into her presentation, Dr. McIlraith described their novel algorithm, which continually updates the robot’s state knowledge through a perceive-plan-act loop. This process involves sophisticated plan recognition algorithms capable of identifying human intentions and goals based on observed actions. Dr. McIlraith underscored the effectiveness of regression techniques employed in this framework to evaluate plan validity from the perspectives of multiple agents. She also elaborated on how their robot, Pepper, used this method in practice to proactively detect and resolve discrepancies between its own and human beliefs, either through communicative actions or physical interventions in the environment.

Lastly, Dr. McIlraith provided a comprehensive overview of empirical studies conducted by her team, showing that robots endowed with ToM through epistemic planning significantly outperformed those without such capabilities. The robots were not only perceived as considerably more socially intelligent and helpful by human participants but also demonstrated enhanced operational effectiveness. Dr. McIlraith concluded her talk by emphasizing the broad implications of integrating ToM-enabled robotic systems into complex human environments, highlighting the potential for significant improvements in robotic assistance and social interaction outcomes.

Keynote speaker: Joshua Tenenbaum

Joshua Tenenbaum presented a computational and cognitive science approach to modeling human intuitive theories of other minds, a framework he referred to as “Bayesian Inverse Planning.” Drawing on some of the explanations done earlier by Dr. Saxe, Tenenbaum underscored a central question: “Can we build machine-implementable models that capture how human minds reason about beliefs, desires, and intentions models that are not only quantitatively predictive and explanatory but also robust and general enough to inform new technologies and real-world interventions?”

He explained how these ideas, originating in cognitive science research, can serve as the foundation for cooperative AI machines that think, learn, and act productively with people, whether they are longtime collaborators or first-time acquaintances. Dr. Tenenbaum highlighted probabilistic programming and language models as key tools that, when integrated, allow Bayesian methods to scale from relatively simple, controlled experiments to complex, real-world settings involving deeper recursive reasoning.

By leveraging these advances, Tenenbaum argued that we can begin to engineer AI systems that exhibit many core elements of human social intelligence: from communication and storytelling to norm adaptation, social evaluation, and moral decision-making. He closed by posing broader questions about why our minds evolved these capacities and how such insights should shape the next generation of intelligent machines, tying back to discussions of the co-evolution of minds and theories of minds introduced by other speakers at the workshop.

Some interesting posters…as much as I can remember.

Presenter: Ramira van der Meulen

  1. Modularity of ToM: Contrary to treating ToM as a dedicated module, the authors argued for a distributed system reflecting multiple cooperative processes, aligning more closely with contemporary cognitive science literature.
  2. Universal Necessity of Advanced ToM: Not every social interaction requires complex ToM; simpler heuristics and scripts often suffice. However, AI should maintain vigilance for scenarios that diverge from expected patterns, necessitating deeper ToM reasoning. Additionally, returns from increasingly higher levels of ToM diminish significantly beyond a certain point, indicating a balanced and task-specific implementation is more efficient.
  3. Homogeneity of ToM: All entities interact uniquely; hence, anthropomorphizing AI by directly mapping human cognition onto it can mislead developers. Effective human-AI collaboration requires mutual understanding, necessitating either educating humans about AI reasoning (‘Theory of AI Mind’) or explicitly designing AI behaviors comprehensibly.
  4. Misconceptions Regarding Current AI’s ToM: While contemporary LLMs show impressive performances on tests traditionally measuring human ToM capabilities, their results partly reflect exposure to training datasets. Empirical exploration indicates that the capacity to simulate ToM through data exposure differs fundamentally from actual, situated ToM used in real-world interactions.

Two fun game-based posters:

Turning game: Game that is based on 3 players talking in a chat room and 1 of them is an AI and the human players need to determine who the AI is. Roughly 40% (can’t totally remember the number) were unable to determine who the AI was. 

Wavelength: In the game, a random object is selected and each of the two players must rank how valuable they think that object is to the other person. The players earn points based on how closely their rankings match. When two strangers play together, their average score is around 5 points. However, when a human plays with an AI, the score rises to about 8-9 points, and when close friends or family members play together, the score averages around 13 points. This suggests that the AI is better at quickly estimating the value another person might assign to an item, even if they’ve just met, compared to two strangers playing together.

DILab at UNESCO’s Digital Learning Week

The Design Intelligence Lab is thrilled to announce that one of its projects, Jill Watson, was selected for presentation at UNESCO’s Digital Learning Week, held from September 2-5, 2024 at UNESCO Headquarters in Paris, France. On September 4th, DILab member Pratyusha Maiti showcased Jill Watson during a breakout session on system-level and teacher-facing generative AI tools, moderated by Mr. Saurabh Roy, Senior Project Officer, Section for Teacher Development at UNESCO.

The presentation highlighted the innovative deployment of Jill Watson, a virtual teaching assistant powered by OpenAI’s ChatGPT 3.5 Turbo. Designed for use in higher education, technical/vocational training, and adult learners in online education, Jill Watson addresses key challenges such as self-directed learning and the need for a strong teaching presence in online environments. The tool integrates with Learning Management Systems (LMS) and employs retrieval-augmented generation (RAG) to provide accurate, contextually relevant responses based on course materials. Jill has been deployed across several courses and institutions and serves to provide an engaging learning environment for learners of varying demography, fostering deeper student engagement, supporting independent learning, and enhancing the overall educational experience. 

Discussions at the Digital Learning Week explored how digital tools can align with climate-friendly educational practices, emphasizing a “twin transition” towards greener, more human-centered learning environments. Discussions also included global AI regulations, with contrasting perspectives from the US and the EU, and the development of AI competency frameworks aimed at fostering critical thinking, inclusivity, and collaboration among learners and educators. The event also spotlighted challenges, including connectivity gaps, cultural shifts in education, and the ethical considerations surrounding AI’s use in marginalized communities. UNESCO’s ongoing efforts, like the Gateways Initiative, highlighted the global push to ensure equitable access to digital learning and sustainable educational practices​.

Full Event Program

ToMinHAI 2024: 1st Workshop on Theory of Mind in Human-AI Interaction

by Qiaosi (Chelsea) Wang (Georgia Institute of Technology, US), Sarah Walsh (Georgia Institute of Technology, US), Mei Si (Rensselaer Polytechnic Institute, US), Jeffrey O. Kephart (IBM Research AI, US), Justin Weisz (IBM Research AI, US), Ashok Goel (Georgia Institute of Technology, US)

“How does trust build between people? It is an offering and a receiving… It is the reaching out between people, laughing at the same moment. It is building a model of the other person inside yourself, placing them in the palm of your hand, rotating them and saying: Yes, I see the flaws… And it is saying: I would rather trust you than be alone.” — Naomi Alderman, The Future“

In psychology, Theory of Mind (ToM) refers to peoples’ ability to attribute mental states such as knowledge, emotions, goals, and beliefs to oneself and others. ToM helps us understand that these mental states may differ from our own. Given the increasing popularity of large language model (LLM)-based conversational agents, this concept has become highly relevant in both the human-computer interaction and machine learning communities.

Theory of Mind plays a fundamentally important role in human social interactions and many researchers have been working on methods to equip AI with an equivalent capability. The hope is to build highly-social, intelligent AI agents that can communicate with people on their level. Simultaneously, researchers are also interested in how people perceive conversational AI systems and their tendencies to attribute mental states, emotions, and intentions to it. These two perspectives on theory of mind are forming an emerging paradigm of Mutual Theory of Mind (MToM) in human-AI interaction in which both the human and AI each possess some level of ToM-like capabilities.

We recently held the first workshop on Theory of Mind in Human-AI Interactions at CHI 2024 to bring together different perspectives on ToM in human-AI interaction and define a unifying research agenda on the human-centered development of Mutual Theory of Mind (MToM). Our workshop focused on three core questions:

  • How to design and build a ToM-like capability for AI systems?
  • How to understand and shape peoples’ mental models of AI?
  • What are the consequences of building mutual theory of mind in human-AI interaction?

These questions were addressed across 15 papers presented in three sessions, as well as in 8 posters, a panel discussion, and a group activity to identify grand challenges in Theory of Mind for human-AI interaction.

Challenges, Opportunities, and Directions of Theory of Mind in HAI (Paper Session I)

Although Theory of Mind (ToM) is a well-known idea in the cognitive and social sciences, its use as a theoretical lens to study human-AI interaction is not yet well established. Human-centered AI researchers are still developing a shared understanding of foundational issues such as what precisely is a “Theory of Mind” in the context of AI systems and what difference it makes in the design of human-AI interactions. The first session comprised five papers that addressed some of the foundational issues in Theory of Mind in human-AI interaction:

Pafla et al. consider three accounts of ToM common in the cognitive and social sciences:

  • Theory Theory, where humans develop a theory of mind to understand the mental states of others,
  • Simulation Theory, where humans imagine themselves to be in the situation of the other when understanding their mental states, and
  • Perception Theory, where perception directly indicates the mental states of others without requiring inferencing.

They note the contradictions between these three accounts and ask for a resolution of the contradictions. They align their paper with Perception Theory, but to resolve the contradictions, they extend it to include “smart” perception in which the mind constructs the subjective experience of mental states based on context. The authors argue that this extended “smart” Perception Theory has four major implications: negotiation of reference between agents, emergence of ToM from social interactions rather than pre-definition, a focus on non-conceptual intentions, and flexible representations of objects. It is interesting to note that this characterization of ToM questions the notion of ground truth in AI because instead of being pre-defined, truth now emerges out of social interactions.

Wang and Goel propose a framework for the development of a mutual theory of mind within human-AI communication. This framework emphasizes the importance of both the content of each party’s models and the processes by which those models are constructed and recognized.

Regarding content, a mutual theory of mind is characterized by three elements:

  • interpretation, in which humans and AI agents can each construct and revise their interpretations of each other based on feedback from the other party,
  • feedback based on interpretations of each other, and
  • mutuality, in which humans and AI agents mutually shape each other’s interpretations of one another through feedback.

Regarding process, the framework suggests that a mutual theory of mind develops in three stages:

  • AI’s Construction of its ToM,
  • User’s Recognition of AI’s ToM, and
  • AI’s Revision of its ToM.
The three stages of development of MToM: Construction, Recognition, and Revision. Figure reproduced from Wang & Goel (2024).

Throughout these three stages, the three content elements of MToM — interpretation, feedback, and mutuality — interact with each other to shape the communications between the human and the AI. In their paper, Wang and Goel briefly describe two empirical studies pertaining to the construction and recognition stages. They conclude by proposing a research agenda for investigating the three stages in the development of MToM.

Ackerman and Shihadeh take a provocative stance that large AI models, such as large language models (LLMs), can be viewed as “a portal to humanity’s collective mind,” reflecting humanity’s collective unconsciousness, including its fallacies and biases. They posit that large AI models are a manifestation of Carl Jung’s notion of a “Collective Unconsciousness,” the idea that humanity shares a collective psyche that encompasses our highest virtues and our deepest prejudices. Ackerman and Shihadeh argue that framing large AI models as representations of our collective unconsciousness offers us the opportunity to reflect upon the “darker aspects” of our collective society and of ourselves. By framing human-AI interaction as engaging with humanity’s collective mind, individuals might recognize that the biases seen in the AI are also found in themselves. This perspective may foster deeper emotional engagement by encouraging individuals to view the computer not as a machine or a human but as a vessel for humanity’s collective subconscious.

In her paper, Street asserts that if LLMs possess a ToM, or if they acquire one, the potential impact could be profound, especially in regard to the alignment problem — how we design and deploy AI systems that behave in accordance with human values. She suggests that at the level of individual users, LLMs with ToM could support human-AI interaction in three ways:

  • Goal specification, by taking the potentially ambiguous goals of humans and defining them in a way that an AI system can achieve them,
  • Conversational adaptation, by tailoring what an AI system says and how it says it on the basis of the inferred mental states of their human interlocutors, and
  • Empathy and anthropomorphism, by facilitating a deeper understanding of users and providing more empathetic responses to them.

Looking beyond individual interactions, LLMs with ToM may facilitate group interactions as well through:

  • Collective alignment, by aligning LLM outputs with social values,
  • Fostering cooperation and competition among humans within the group, and by
  • Assisting groups with moral reasoning and collective decision making.

Street suggests that research agendas for LLMs with a ToM be inclusive of interactions at both the individual and group levels.

Finally, Weisz et al. presented three design fictions that probe the potential consequences of operationalizing a MToM between human users and one or more AI agents. The first design fiction explores a utopian vision of an operationalized MToM in which an AI agent is capable of learning about a user and predicting their behavior. This story highlights the beneficial outcomes that MToM may bring to workers within an organization and how MToM might shape the future of work:

  • helping us identify and focus on the tasks we truly enjoy,
  • providing a buffer from coworkers to improve our ability to achieve and maintain flow,
  • proactively filling in knowledge gaps,
  • improving social connectedness, and
  • helping us focus on our higher-level work goals.

The second and third design fictions investigate dystopian visions of operationalizing MToM. In the second fiction, a human interacts with a collection of bots, where each bot constructs its own user model of the human based on the purpose of the particular bot. This story considers cases in which bots with different user models make unclear transitions and exchange incomplete information about the human and their needs. In the third fiction, the AI’s model of the human user is so good that the human comes to completely rely on the AI agent and then applies the AI to a domain with which it is unfamiliar. Weisz et al. conclude their paper with a discussion of several research issues the above design fictions raise, including the need for predictive models of human users and the importance of explanations for helping users calibrate their trust with AI agents.

Theory of Mind in Human-AI Collaboration (Paper Session II)

An agent is an effective collaborator if, through its interaction with humans, it improves the speed with which human objectives are attained or the degree to which they are realized. The papers in this session addressed how an AI agent could infer a person’s intent or mental state and leverage that understanding in a helpful way. These papers offered techniques for agents to infer human intents through a combination of observed actions, utterances, and physical manipulations. They also examined methods for agents to communicate their belief states about humans and examined how that affects the human’s performance on a task. Finally, one paper offered practical heuristics and algorithms for improving the feasibility of real-time collaboration given computational constraints.

Although the papers are listed above in the order of presentation, we begin our synopsis with the work of Shi et al., which focused on inferring intent by observing a series of actions in an environment. They demonstrated an approach that combines the ability of a deep neural network (DNN) to handle complex action spaces with the ability of Bayes methods to represent and cope with uncertainty. As shown below, they trained a DNN to take a sequence of human actions and produce a probability distribution of next possible actions. They then fed those distributions to a separate Bayesian model to generate a probability distribution over human intentions.

Figure reproduced from Shi et al. (2024)

Shi et al. tested their technique on two data sets and found that, in most cases, the true intent was identified as the most likely one, especially as the length of the observed action sequence was increased.

Zhi-Xuan et al. explored the idea of predicting human intent even further by considering both actions and linguistic utterances. Their technique, called Cooperative Language-Guided Inverse Plan Search (CLIPS), models humans as cooperative planners who communicate joint plans through language that may be somewhat ambiguous in nature. The authors evaluated their technique in a “Doors, Keys & Gems” environment in which a human and an AI agent (depicted as a robot) move through a maze to collect gems. The human is able to delegate tasks to the robot, such as fetching the differently-colored keys to open the doors.

In the example below, when the agent only observes the movement of the human’s figure (depicted with arrows within the tiles), it might infer that the human is moving toward the blue door, concluding that it should fetch the blue key. However, when the human says “Can you pass me the red key?,” the agent is able to combine this instruction with its observation of the human’s movement and infer that the human’s plan is to open the red door.

Figure reproduced from Zhi-Xuan et al. (2024)

The authors demonstrated through ablation studies that combining actions with utterances is far more accurate in predicting the human’s plans than using either modality alone. Interestingly, CLIPS even outperformed human observers who attempted to infer a player’s intent from their actions and utterances.

AI agents can also proactively initiate communications to improve human-agent collaboration. Ying and Gajos ran a study using a simulated grocery shopping scenario where participants were asked to shop for ingredients to make a specific recipe, and an AI agent observed the ingredients selected by the participant and inferred what recipe the participant was attempting to make. Their results showed that when the agent conveyed its belief about the participant’s goal and how confident it was in that belief, participants were more likely to accept the agent’s ingredient recommendations. Thus, when the agent was transparent about its beliefs, participants were able to calibrate their trust in its recommendations. The authors examined two different ways for the agent to communicate its beliefs: a “Show” approach in which the assistant conveyed its belief using words (e.g. “I’m certain that you are shopping for chicken soup”) and a “Tell” approach in which the assistant conveyed its belief using numbers (e.g. “Chicken soup, confidence 80%”). The authors reported that both forms led to improved trust calibration, although the “Tell” approach also reduced the amount of time to complete the shopping task.

Another way to convey one’s mental state is through the physical manipulation of objects. Sidji et al. conducted studies of people engaging in Hanabi, a cooperative game involving partial information and restricted communication. The authors used eye-tracking footage of people playing Hanabi and found that players externalized their intentions and beliefs by rotating, reordering, and reconfiguring the cards held in their hand. Thus, physical orientation created a side-channel to convey one’s beliefs to other players (and to one’s self, as a memory aid). This work raises an interesting prospect for multimodal agents (especially robots that have a physical body) to observe and interpret non-verbal human actions as an additional mode of communication.

An important practical question is whether techniques used to infer and act upon human intent can be applied in real time in complex real-world environments. Schröder and Kopp pointed out that approaches such as LLMs and Bayesian Theory of Mind (BToM) are too slow to be viable in online settings. Instead, they proposed an approach that entails action-driven and resource-sensitive inference of a human’s mental state. In this approach, an AI agent has access to a variety of BToM models of varied cost and fidelity, and strategically switches amongst the models based on computational availability and time sensitivity. They also focus the models to consider only those potential actions that are most likely to be relevant on the basis of spatial proximity or shared artifacts. To test these ideas, the authors are developing an agent architecture that supports human-agent collaboration in a multi-player video game called “Overcooked!”, in which each player controls a chef and attempts to collaboratively prepare meals in response to incoming orders.

Theory of Mind for Socially Intelligent AI (Paper Session III)

Theory of Mind (ToM) refers to attributing mental states such as intentions, goals, emotions, and beliefs to oneself and others. This capability is essential for understanding and predicting behavior in social interactions. In the context of AI, ToM enables machines to comprehend, predict, and respond to the mental states of humans and other AI agents, thereby enhancing their social intelligence. The five papers in Session III collectively explored applications of ToM in human-AI interaction. They examined how AI systems can be designed to better understand and respond to human mental states, thereby improving communication, collaboration, and personalized support during learning, life transitions, and everyday activities. Additionally, the papers touched upon the broader implications of artificial minds and the role of ToM in shaping interactions between children and social interactive agents.

Doll and Si delve into the significance of rapport in human-AI interactions, emphasizing the integration of Theory of Mind (ToM) to improve these interactions. The paper argues that AI systems equipped with ToM not only foster better communication but also enhance collaborative efforts. Through a comprehensive review of both human-human and human-AI relationship studies, the authors highlight how empathetic and adaptive responses from AI can establish a deeper connection and trust with users. Rapport is crucial not only in personal and professional domains but also in educational settings, where AI’s understanding of human emotions and intentions plays a pivotal role in facilitating effective interactions and achieving shared goals in mixed human-AI teams.

Wang, Li, Zhou, and Goel explore how Theory of Mind (ToM) can be used to personalize AI assistance during significant life transitions such as career changes or retirement. The paper examines the integration of ToM capabilities in AI to understand and support individuals’ emotional and psychological needs during these periods. It highlights the expansive potential of AI to infer mental states and align its operations with users’ needs, offering more natural and intuitive interactions. Despite this potential, the paper stresses the fragmentation in current applications and the necessity for a comprehensive exploration to guide AI development that genuinely understands and responds to the intricate web of human mental states.

Asthana and Collins-Thompson present a literature review focused on applying Theory of Mind (ToM) to educational settings, with a specific emphasis on generative AI. They discuss how AI can be designed to better understand and adapt to students’ learning styles and mental states, thereby improving educational outcomes. Their review emphasizes the potential of generative AI to effectively diagnose and address students’ knowledge gaps. Enhanced ToM capabilities in AI could significantly improve personalized learning experiences, offering dynamic and responsive educational support tailored to individual students’ cognitive and emotional needs. The paper also highlights future opportunities for developing AI tools that can perform complex educational tasks with greater autonomy and adaptability.

The presentation “Weaving a Theory of Artificial Minds” by Bharadwaj and Dubé expands on the concept of a Theory of Artificial Minds (ToAM), drawing from extensive literature on children’s interactions with AI and robots. Studies consistently show that children anthropomorphize AI systems by attributing human-like emotions, intentions, and personalities to them, especially when these systems have human-like voices and interactive features. Younger children tend to view AI systems as “alive” or possessing life-like qualities, though this perception diminishes as they grow older and their understanding of biological life becomes more sophisticated. According to both Media Equation Theory and the Computers as Social Actors (CASA) paradigm, people, including children, treat computers and AI as social entities and respond to them as they would to humans. Many studies have observed children showing emotional and social engagement with AI that exhibits social cues. Further, some studies suggest that children perceive AI systems as a distinct ontological category, neither fully human nor fully machine, indicating that engagement levels vary based on how children categorize the AI entities they interact with. These studies highlight the importance of Theory of Mind (ToM), as children with more advanced ToM skills are better able to navigate and utilize AI systems effectively. Based on their literature review, Bharadwaj and Dubé identify the need for a dedicated theoretical model of how humans understand AI systems. Their proposed Theory of Artificial Minds (ToAM) framework seeks to integrate insights across various fields to explain how children and adults perceive and interact with AI and guide the development of AI systems that are more attuned to human social norms and expectations.

Finally, Arbelo et al. examine young children’s interactions with Social Interactive Agents (SIAs) in educational settings, including Alexa, MIKO, and a conversational virtual assistant they developed called Puntal. Utilizing a qualitative case study approach, the researchers observed how these technologies influenced the learning and engagement of early childhood education students from two schools in Tenerife, Spain. They found that Puntal was particularly effective due to its versatile functionality, including an automatic translation feature that facilitated inclusive communication in multicultural classrooms. Their study highlights the significant potential of SIAs to enhance educational experiences by providing interactive and tailored educational support.

Alexa, Puntal and MIKO. Figure reproduced from Arbelo et al. (2014).

Poster Session

The poster session consisted of preliminary research efforts that spanned all of the topics of the workshop. Six posters were presented in this session and authors presented their work in 90 second lightning talks.

Rogers and Scott presented their ideas on how predictive brain theory explains how humans and AI agents may establish a mutual theory of mind. . Predictive brain theory posits that the predictive brain aims to reduce the number of surprises it encounters. When a person (or AI agent) makes an error in predicting the behavior of another entity, they may either update their model of the world or take an action so that the world better fits their model. They offered a small case study showing how the two possibilities may manifest in a human-AI interaction when a predictive error is made. In the first case, a ToM-enabled model may ask the user follow-up questions to better understand their beliefs and goals. In the second, the AI agent asserts that its model of the user is accurate, which comes across as arrogant and undermines the user’s own understanding of themselves.

Otenen discussed the critical role of developing Mutual Theory of Mind in memory-aiding technologies. She outlined three design directions:

  1. The need to understand a human’s theory of mind of AI, as it influences the content and length of the human-AI interaction;
  2. The need to design human-AI interactions for people of varying levels of ToM to make AI more accessible and equitable; and
  3. The need for understanding theory of mind in AI systems and how such systems can dynamically adapt to a user’s emotional needs.

Wester et al. identified how self-presentation styles impact responses from an LLM.They examined several ways a human could formulate a request for help with a homework problem, with the LLM providing differing levels of detail to each request.

Several posters focused on ToM in human-AI collaboration. Hirschmann et al. explored the potential of an AI ToM capability to enhance human-robot collaboration in a cooperative game scenario. They examined how the complexity of AI decision-making and the minimization of ambiguity impacted collaboration. They found that agents which used simpler decision-making strategies, but who sought to minimize ambiguity, matched or exceeded agents that used more complex decision-making logic. These results highlight the importance for AI agents to make their behaviors clear and unambiguous to human collaborators.

Tsirtsis et al. examined how people assign responsibility to a human vs. an AI agent in a semi-autonomous driving environment. They proposed a model of responsibility that considers the expectedness of an agent’s action and the consequences of alternative actions.

Narayanan and Feigh identified factors that influence shared mental models in human-AI teams, with a focus on contexts in which decisions are made in a chain-of-command fashion. Through several illustrative examples of search and rescue missions conducted by human-AI teams, they emphasize the importance of studying the influence of decision hierarchies on the elicitation, development, and maintenance of shared mental models to achieve effective and optimal human-AI team performance.

Invited Panel on Theory of Mind in Human-AI Interaction

After the paper and poster sessions that showcased theoretical and empirical work on Theory of Mind in Human-AI interaction, we had a wonderful panel that brought everything together and sought to answer one key question:

What is the role of Theory of Mind in Human-AI interaction?

Our panel featured researchers from academia and industry research:

  • Yvonne Rogers, Professor of Interaction Design at University College London,
  • Tanya Kraljic, Staff UX Researcher at Google Research, and
  • Justin Weisz, our fellow workshop organizer and Research Manager and Senior Research Scientist at IBM Research.

The panel was moderated by Qiaosi (Chelsea) Wang, the lead organizer of our workshop.

During the discussion, the panelists highlighted the various roles that Theory of Mind could play within human-AI interaction. Yvonne raised the idea of using Theory of Mind to help inform the design of human-AI interaction. Decades of research in psychology have examined the role that Theory of Mind plays in human-human interactions. Yvonne challenged us to think about how this body of work could be extended to shape the design of human-AI interactions? Justin and Tanya echoed this point and suggested Theory of Mind as a means to an end rather than the end itself. In addition, they argued that the concept of “Theory of Mind” manifests differently across different academic disciplines. For example, HCI researchers often talk about “mental models,” whereas AI & machine learning researchers talk about “user modeling.” The panelists reflected on whether these differences in language use were getting in the way of establishing common ground, and they also advocated for focusing more on how the concept of “Theory of Mind” — no matter how it is labeled — can be used to enhance human-AI interactions.

The next topic the panelists discussed was how Theory of Mind could be used to inform the design of AI tools and products in human-AI communication. Justin drew on his experiences as a father of young children, where he has an understanding of the words his children do and do not yet have in their vocabulary, and uses it to introduce new words they don’t yet know. He argued that AI systems equipped with a ToM can have similar models of an individual’s knowledge, enabling them to communicate with users at an individual level. For example, a software engineer could ask a ToM-equipped AI assistant to explain a piece of source code, and that assistant would be able to produce an explanation at their level of comprehension. On a similar note, Tanya then talked about how equipping AI with ToM could help people interact more successfully with AI assistants. She pointed out that given the current design of AI assistants with opaque working mechanisms, the responsibility is put on users to figure out what the AI assistants can do. A lot of times, if the assistant doesn’t provide good-quality responses, some degree of blame lies in how it was prompted by the user. Tanya suggested that the idea of ToM can be powerful in considering how to enable AI systems to explain its working mechanism tailored to individual users to help users interact more successfully.

The final topic discussed by the panelists was the role of ToM in designing responsible and human-centered AI systems. Tanya pointed out that many AI technologies claim to have Theory of Mind capability when they actually do not. This type of claim only confuses people and leads to very high expectations of AI technology. Tanya suggested that it is part of our responsibility to be transparent about AI systems’ true capabilities. Yvonne and Justin also highlighted that limiting AI’s ToM capabilities in certain dimensions or its application contexts could be helpful in designing responsible AI systems. For example, Yvonne pointed out that we should question when ToM capabilities are or are not appropriate and useful. She also highlighted how the purpose of ToM can go beyond personalization, such as how a ToM-equipped AI could help people reflect and think about themselves and their own behaviors. Justin pointed out that the contexts in which we design ToM-enabled AI systems matter: in cooperative scenarios, a ToM-equipped AI may be beneficial to human-AI collaboration, but in competitive scenarios, a ToM-equipped AI may be able to take unfair advantage of people.

Our panel discussion highlighted how we are at an exciting and unpredictable time in history regarding the rapid development of AI technologies. As HCI researchers, it is imperative that we continue striving toward designing responsible and human-centered ways of incorporating Theory of Mind into the design of human-AI interactions.

Grand Challenges in Theory of Mind for Human-AI Interaction

We conducted a group activity in the afternoon to identify and address grand challenges in Mutual Theory of Mind. Our goal was to identify ambitious, but achievable, challenges that could be used to demonstrate or evaluate MToM. This activity engaged our workshop attendees by focusing their discussion on important technical, societal, and ethical challenges that accompany the development of Mutual Theory of Mind.

The first group activity focused on brainstorming challenges facing the field of Mutual Theory of Mind in human-AI interaction research. Small working groups generated a myriad of ideas, which they then clustered and summarized to the larger group. Many of these challenges were phrased as questions, such as:

  • What are the ethical implications of AI systems having ToM?
  • How should AI ToM be communicated to human users?

Some proposals focused on identifying specific development challenges:

  • Non-anthropomorphic ToM challenge: Can AI have a different type of theory of mind from us?
  • The Deception Challenge: Create an AI that predicts or detects human deception (a benchmark, a goal)

Many philosophical and ethical questions were raised in this session, including:

What are the ethical implications of non-human systems inferring human mental states (in a way that is different from human: human ToM inferences)? How might it impinge on human rights? (e.g. freedom of thought, autonomy)

In the second group activity, teams were asked to ideate on how researchers might approach one of the challenges identified by their team. Participants came up with a variety of approaches that coalesced into four areas: techniques, tools, measures, and theories.

The group activities led our workshop attendees to ponder some of the deeper philosophical questions within MToM in human-AI interaction. It also helped them identify some of the most impactful research directions and challenges in domains where MToM is relevant.

We hope these ideas will provide inspiration and direction to human-centered AI researchers examining issues around mutual theory of mind!

The Return of Jill Watson

Georgia Tech’s Design Intelligence Laboratory and NSF’s National AI Institute for Adult Learning and Online Education have developed a new version of the virtual teaching assistant named Jill Watson that uses OpenAI’s ChatGPT, performs better that OpenAI’s Assistant service, enhances teaching and social presence, and correlates with improvement in student grades. Insofar as we know, this is the first time a Chatbot has been shown to improve teaching presence in online education for adult learners.

JillWatsonReturnsDownload