Joint Human-AI Systems Publications

Joint Human-AI Systems Publications

Joint Activity Testing Implementation Guide: Human-Machine Teaming Handbook

Public Release: 2025

Report Date: July 2025

Authors: Dane A. Morey, Amy Ryder, Jennifer Winner, Mimi Cai, Liana Algarín, Michael F. Rayo, Cody Parker, Priyanka Tewani, Katie Kopren, Jason "Twitch" Green, and Paige Kinney

The Ohio State University. Translational Data Analytics Institute

Abstract

This implementation guide provides a structured methodology for evaluating human-machine teaming (HMT) systems using Joint Activity Testing (JAT). Developed through research conducted for the National Geospatial-Intelligence Agency (NGA), the guide outlines practical approaches for assessing the performance, resilience, and brittleness of human-AI teams across varying operational challenges. The handbook presents a five-phase evaluation framework covering study framing, planning and design, data collection, analysis, and reporting. It also introduces Joint Performance Graphs (JPGs), discusses complementary methods including Joint Activity Design (JAD) heuristics and Decision-Centered Testing (DCT), and provides examples, tools, templates, and appendices to support the evaluation of AI-enabled technologies and human-machine teams.

Evaluation of Joint Human-Machine Systems: Literature Review & Recommendations

Report Date: May 2026

Completed: February 2025

Authors: Dr. Michael F. Rayo, Dr. Dane A. Morey, Dr. Morgan E. Reynolds

Ohio State University. Translational Data Analytics Institute

Abstract

This report presents findings from a literature review conducted by the Cognitive Systems Engineering Laboratory (CSEL) examining testing and evaluation approaches for human-automation teaming (HAT). Drawing from 227 articles, the report identifies key testing and evaluation dimensions, recommends priority approaches for future research, and outlines technology, protocol, and training requirements for the Human-Autonomy Teaming Testbed for Research (HATTER). The report provides guidance for researchers and practitioners developing and evaluating joint human-AI and human-machine systems in complex operational environments.