Joint Human-AI Systems Publications
Joint Activity Testing Implementation Guide: Human-Machine Teaming Handbook
Public Release: 2025
Report Date: July 2025
Authors: Dane A. Morey, Amy Ryder, Jennifer Winner, Mimi Cai, Liana AlgarÃn, Michael F. Rayo, Cody Parker, Priyanka Tewani, Katie Kopren, Jason "Twitch" Green, and Paige Kinney
The Ohio State University. Translational Data Analytics Institute
Abstract
This implementation guide provides a structured methodology for evaluating human-machine teaming (HMT) systems using Joint Activity Testing (JAT). Developed through research conducted for the National Geospatial-Intelligence Agency (NGA), the guide outlines practical approaches for assessing the performance, resilience, and brittleness of human-AI teams across varying operational challenges. The handbook presents a five-phase evaluation framework covering study framing, planning and design, data collection, analysis, and reporting. It also introduces Joint Performance Graphs (JPGs), discusses complementary methods including Joint Activity Design (JAD) heuristics and Decision-Centered Testing (DCT), and provides examples, tools, templates, and appendices to support the evaluation of AI-enabled technologies and human-machine teams.
Evaluation of Joint Human-Machine Systems: Literature Review & Recommendations
Report Date: May 2026
Completed: February 2025
Authors: Dr. Michael F. Rayo, Dr. Dane A. Morey, Dr. Morgan E. Reynolds
Ohio State University. Translational Data Analytics Institute
Abstract
This report presents findings from a literature review conducted by the Cognitive Systems Engineering Laboratory (CSEL) examining testing and evaluation approaches for human-automation teaming (HAT). Drawing from 227 articles, the report identifies key testing and evaluation dimensions, recommends priority approaches for future research, and outlines technology, protocol, and training requirements for the Human-Autonomy Teaming Testbed for Research (HATTER). The report provides guidance for researchers and practitioners developing and evaluating joint human-AI and human-machine systems in complex operational environments.