AHIMR'26 — AI-Mediated Heads-Up Interaction in Wearable Mixed Reality

The shift

From what to display to how an agent decides

As smartglasses, head-mounted displays, and body-worn devices move from research prototypes to mass-market products — Apple Vision Pro, Meta AI Glasses, Android XR — the central design challenge in wearable mixed reality is no longer what to display. The harder question is how an AI agent should decide what to surface, in what form, and when, on behalf of a user whose attention is divided between physical and digital contexts.

AHIMR'26 focuses on AI-mediated interaction in mixed reality: systems where an AI agent, rather than the user, decides how information is transformed, timed, and delivered through wearable MR devices. This is the defining mechanism of the Heads-Up Computing paradigm — the system adapts to the user's state rather than the user adapting to the device — grounded in resource-rational HCI, where human attention is a finite resource to steward rather than exploit.

Topics of interest

Three interdependent challenges

Every AI-mediated MR system must solve all three. We organize the workshop — and the call — around them.

Cluster 1

Sensing & Modelling

How the AI knows the user's state

Real-time attention & cognitive load estimation from commodity wearable sensors
Context and physical activity recognition for adaptive MR interfaces
Computational models and simulations of user behaviour for wearable MR
Multimodal user state inference: gaze, voice, IMU, physiological signals

Cluster 2

Transformation & Delivery

How the AI re-authors & times information

Conversational and multimodal information access for wearable MR
Intelligent retrieval and summarisation for heads-up interaction
Context-aware search and recommendation in MR environments
Multimodal output selection: voice, gaze, peripheral display, haptics
Proactive AI assistance and knowledge surfacing on wearable devices
Delivery optimisation under divided attention: timing, urgency, form, density

Cluster 3

Trust, Agency & Ethics

What it means for AI to act on your behalf

User trust and control when AI suppresses, modifies, or delays information
Social acceptability and privacy of AI-mediated wearable MR in public
Human agency and oversight in AI-mediated MR systems
Evaluation methodologies for real-world deployment

Invited keynote

Mark Billinghurst

We open AHIMR'26 with a keynote from Mark Billinghurst, a leading figure in augmented reality, wearable computing, and empathic computing.

Monday, October 5 · 9:10–9:50 CEST · 15:10–15:50 HKT

Research Directions in AI-Mediated Heads-Up Interaction

Mark Billinghurst · Professor of HCI, University of Auckland & University of South Australia

After decades of promise, Smartglasses are beginning to enter the mainstream. For the first time millions of people will be wearing display and audio enhanced glasses in everyday life. The addition of AI to this wearable platform opens up new possibilities for transforming interaction, information presentation and collaboration, while addressing concerns around privacy and ethics. This presentation gives an overview of some of these opportunities, and the research directions that will need to be addressed to enable Smart Glasses to become mainstream.

30-minute talk followed by 10 minutes of Q&A · Open to all workshop attendees, in-room and online. Mark directs the Empathic Computing Lab, with 300+ papers across wearable computing, AR, and mobile interfaces; he is a Fellow of the Royal Society of New Zealand and a recipient of the IEEE VR Technical Achievement Award.

Half-day · Monday, October 5, 2026 · 9:00–13:00 CEST (Bari)

A working session, not a mini-conference

Twelve accepted papers run in four thematic sessions of three, and every paper gets the same 10-minute slot — 8 minutes to present plus 2 minutes of Q&A. The morning closes with world-café breakouts that produce written open-problem statements.

9:00–9:10

Opening & Framing

The three open problems of AI-mediated heads-up interaction, and how the morning is structured around them.

9:10–9:50

Keynote — Mark Billinghurst Invited

30-minute keynote followed by 10 minutes of moderated discussion: Research Directions in AI-Mediated Heads-Up Interaction.

9:50–10:20

Session 1 — Sensing, Reasoning & Evaluation 3 papers · 30 min

Chair · Ian Oakley

How we sense the user, model their state, reason about what to do, and measure whether any of it works.

P6Exploring Deliberative Reasoning for Agentic AR Navigation
P16A Dual-Pathway Knowledge Framework for Cognitive-Aligned Interaction in Wearable Mixed Reality
P10You Cannot Optimize What You Cannot Measure: Multitasking Evaluation as the Missing Foundation

10:20–10:50

Session 2 — Transformation & Delivery 3 papers · 30 min

Chair · Nuwan Janaka

What the agent renders, and through which channel.

P7AI-Mediated Adaptive Content Delivery for Contextual Learning in Everyday AR
P15Toward Scene-Adaptive Text Presentation for AI-Mediated Heads-Up Interaction
P9Toward AI-Supported Audio-Based Situated Analytics

10:50–11:05

Coffee Break

11:05–11:35

Session 3 — Agents, Affect & Mediation 3 papers · 30 min

Chair · Hai-Ning Liang

How agents plan and act on your behalf, how they carry emotion, and how they decide what reaches you.

P12Heads-Up Agentic Interactions: Opportunities and Challenges
P11EarCoach: AI-Mediated Spoken Scaffolding for Heads-Up Programming Support
P8Kaomoji as Resource-Rational Affective Mediation

11:35–12:05

Session 4 — Users, Access & Inclusion 3 papers · 30 min

Chair · Haiming Liu

Who the system is for, and what it costs — in attention, motor effort, and predictability — to reach information.

P5Advancing GenAI VR with Multimodal Interfaces for Older Adults
P13Toward AI-Mediated Adaptive MR Guidance for Older Adults' Movement Learning
P14Toward AI-Mediated Candidate Delivery for Chinese Pinyin Input in VR

12:05–12:35

World-Café Breakouts 3 parallel groups

One open problem per group: reliable attentional sensing; automating transformation across modalities; and the delivery objective function. Each group leaves with a written problem statement.

12:35–12:55

Rapporteur Reports & Synthesis

Three 3-minute reports from the breakout groups, followed by a moderated synthesis that names the cross-cutting priorities.

12:55–13:00

Closing

Next steps, and the road toward a special-issue seed document.

Every paper gets the same 10 minutes (8 + 2 Q&A), and the 2-minute block also absorbs speaker handover, so sessions run back-to-back. All times are CEST (Bari); for Hong Kong, add 6 hours.

Hybrid participation

Join online

The workshop runs hybrid from Bari. Remote attendees and remote speakers join the same session over CityU Zoom.

One Zoom room for the whole morning

The same link covers the keynote, all three paper sessions, the breakouts and the closing synthesis. Please join a few minutes early so we can admit you from the waiting room.

Open the Zoom room

Remote speakers: please join 10 minutes before your session for a short tech check, and have your slides ready to share.

Zoom linkhttps://cityu.zoom.us/s/8123777600
Meeting ID812 377 7600
Bari (CEST)Monday, October 5 · 9:00–13:00
Hong Kong (HKT)Monday, October 5 · 15:00–19:00
Online questionsChat and raised hands are relayed to the room during every Q&A

Accepted papers

Twelve papers, four sessions

All accepted papers appear in the ISMAR 2026 Adjunct Proceedings on IEEE Xplore. Every paper — full, short, or position — receives the same 10-minute presentation slot (8 minutes + 2 minutes Q&A).

Session 1 — Sensing, Reasoning & Evaluation 9:50–10:20 CEST

How we sense the user, model their state, reason about what to do, and measure whether any of it works. Chair: Ian Oakley.

P6Full paper

Exploring Deliberative Reasoning for Agentic AR Navigation

Robert Hoeh · Jalal Uddin · Gabriel Coffee · Jemimah Ogbomoh · Peeranat Wattananuruk · Yu Zhao

P16Full paper

A Dual-Pathway Knowledge Framework for Cognitive-Aligned Interaction in Wearable Mixed Reality

Amit Kumar Jaiswal · Haiming Liu

P10Full paper

You Cannot Optimize What You Cannot Measure: Multitasking Evaluation as the Missing Foundation of AI-Mediated Heads-Up Interaction

Nuwan Janaka · Runze Cai · Yang Chen · Chenyu Zhao · Shengdong Zhao

Session 2 — Transformation & Delivery 10:20–10:50 CEST

What the agent renders, and through which channel. Chair: Nuwan Janaka.

P7Full paper

AI-Mediated Adaptive Content Delivery for Contextual Learning in Everyday AR: A Resource-Rational Design Exploration

Lixiang Zhou · Weisheng Zhou · Hong Su · Tao Zhang · Liubing Zheng · Yang Tian

P15Full paper

Toward Scene-Adaptive Text Presentation for AI-Mediated Heads-Up Interaction

Zhigang Ni · Lei Yang

P9Full paper

Toward AI-Supported Audio-Based Situated Analytics: Open Challenges and Pitfalls Beyond Multi-Story MR Navigation

Mohammad Javadian Farzaneh · Emre Yildirim · Faraz Badali Naghadeh · Selim Balcisoy

Session 3 — Agents, Affect & Mediation 11:05–11:35 CEST

How agents plan and act on your behalf, how they carry emotion, and how they decide what reaches you. Chair: Hai-Ning Liang.

P12Full paper

Heads-Up Agentic Interactions: Opportunities and Challenges

Shengdong Zhao · Hai-Ning Liang · Haiming Liu · Nuwan Janaka · Ian Oakley

P11Short paper

EarCoach: AI-Mediated Spoken Scaffolding for Attention-Aware Heads-Up Programming Support

Jai Sachdeva · Prashant Rawat

P8Full paper

Kaomoji as Resource-Rational Affective Mediation: A Design Framework for Low-Intrusion Wearable AI Companions

Weisheng Zhou · Yufei Ren · Zihui Huang · Xiaofeng Liang · Jiayi Zhao · Yang Tian

Session 4 — Users, Access & Inclusion 11:35–12:05 CEST

Who the system is for, and what it costs — in attention, motor effort, and predictability — to reach information. Chair: Haiming Liu.

P5Position paper

Advancing GenAI VR with Multimodal Interfaces for Older Adults

Paulina Becerril Palma · Yuki Onishi · Frode Guribye · Miroslav Bachinski

P13Full paper

Toward AI-Mediated Adaptive MR Guidance for Older Adults' Movement Learning: A Formative Study with Interviews and Prototyping Workshops

Tong Qian · Jiaxin Zhang

P14Short paper

Toward AI-Mediated Candidate Delivery for Chinese Pinyin Input in Virtual Reality

Xinru Cheng · Yushi Wei · Hai-Ning Liang

Why it matters

Objectives & expected outcomes

Objectives

Establish AI-mediated interaction as a first-class research topic within ISMAR, distinct from prior AR and mobile HCI framings.
Surface and stress-test the three open problems blocking progress: user-state sensing, AI-automated transformation, and the delivery objective function.
Create a shared vocabulary and evaluation criteria researchers can use to compare and build on each other's work.
Connect MR systems research with HCI, AI agents, interactive IR, multimodal interfaces, cognitive modelling, and wearable computing.

Expected outcomes

Three written open-problem statements — one per breakout group — published as a community document after the workshop.
A set of IEEE Xplore-indexed papers establishing the field's current state.
A shared research agenda identifying the most tractable next steps for AI-mediated MR.
A seed document toward a future TOCHI or IEEE TVCG special-issue call.

Participation

Submissions are closed

Thank you to everyone who submitted — we received 16 submissions and accepted 12. All accepted papers appear in the ISMAR 2026 Adjunct Proceedings on IEEE Xplore.

Attend in person in Bari, or join the workshop remotely over CityU Zoom.

Presentation format

Accepted papers were reviewed by the program committee, and camera-ready versions use the IEEE Computer Society VGTC conference format (tc.computer.org/vgtc/publications/conference). All accepted papers appear in the ISMAR 2026 Adjunct Proceedings on IEEE Xplore.

Every paper gets the same 10-minute slot, and chairs keep strict time so each session finishes on schedule; a visible countdown runs in the room and on the Zoom share.

12 papers

Four sessions of three

10 min each

8 min talk + 2 min Q&A

Questions? Contact shengdong.zhao@cityu.edu.hk.

Key dates

CfP availableJun 19, 2026

Paper submission deadline (closed)Jul 10, 2026

Acceptance notificationJul 17, 2026

Camera-ready dueJul 31, 2026

Workshop — Bari & onlineOct 5, 2026

All deadlines were 23:59 Anywhere on Earth (AoE). The workshop runs 9:00–13:00 CEST (15:00–19:00 HKT).

Organizers

Who's running it

A cross-institutional team spanning Heads-Up Computing, AR/MR systems, wearable input, and interactive information access.

Shengdong Zhao

Professor

City University of Hong Kong

Pioneered the Heads-Up Computing paradigm and authored its foundational CACM article (2023). 100+ papers at CHI, UIST, and UbiComp, with multiple best-paper awards; co-organized the Heads-Up Computing workshop at UbiComp 2024.

Homepage

Mark Billinghurst

Professor of HCI

Univ. of Auckland & Univ. of South Australia

Directs the Empathic Computing Lab with 300+ papers across wearable computing, AR, and mobile interfaces. Fellow of the Royal Society of New Zealand; recipient of the IEEE VR Technical Achievement Award.

Homepage

Ian Oakley

Professor

KAIST, South Korea

Two decades of wearable technology research, directing the Wearable & Interactive Technology Lab. Work on smartglasses, novel input, and haptics — published widely at CHI, UIST, and IEEE venues; UIST 2025 co-chair.

Homepage

Haiming Liu

Associate Professor

University of Southampton, UK

Specializes in interactive information access, multimodal AI interfaces, and human-centred retrieval — directly relevant where keyboard-and-mouse interaction is unavailable. Published at CHIIR, ICTIR, and ECIR; chairs the CHIIR Steering Committee.

Homepage

Hai-Ning Liang

Associate Professor

HKUST (Guangzhou), China

Researches novel interaction techniques for VR/AR/MR, gaming, and visualisation. Previously Head of Computing at XJTLU; editorial board member for several venues and an active organizer at top HCI conferences.

Homepage

Nuwan Janaka

Postdoctoral Research Fellow

City University of Hong Kong

PhD (NUS) pioneered interruption management for heads-up optical see-through HMD notifications. 15+ papers at CHI, UbiComp, ISMAR, and HRI; Associate Chair for CHI (2024–26) and UIST (2025–26); TOCHI Distinguished Reviewer.

Homepage