AI Evaluation Is Expanding From Performance Alone to Systems for Participation and Safe Completion

New criteria alongside performance

Model performance remains foundational in frontier AI. Evaluation is also expanding to ask who can use that capability, how much real work they can complete safely, and whether people can regain control when something goes wrong. Treating performance, participation, and safe completion as one system is becoming a condition for translating technical capability into real-world value.

Google DeepMind’s announced sign-language-to-text model, SL2T, makes this perspective concrete. It treats sign language not as a set of gestures or “English on the hands,” but as a language with its own grammar and vocabulary, and integrates translation into existing interfaces, Gboard and Live Transcribe. By connecting users to familiar entry points for search, writing, and conversation instead of requiring a specialized environment, the design illustrates how accessibility shapes participation in practical work, not merely access to a feature.

The announced capability is currently limited to ASL-to-English translation, with rollout beginning on Pixel 11. Published examples also show errors, including the mistranslation of “prey” as “grey” during rapid fingerspelling, as well as changes or omissions involving rare signs, tense, passive constructions, and spatial descriptions. Opening an entry point to use is not the same as making every result safe to trust without qualification.

Turning co-design into verifiable governance

According to the announcement, Deaf people and subject-matter experts contributed to SL2T’s conception, data collection, evaluation, and impact assessment, and participate in development priorities through an advisory committee. Designing with affected communities, rather than only for them, is an important direction for participatory governance.

The presence of participation alone does not establish the quality of governance. It should also be possible to trace whose views were represented, how disagreement was handled, and how input changed product decisions. The editorial team considers the ability to intervene in AI processing, correct an output, stop an action, and reverse it where appropriate to be important evaluation criteria. These controls were not demonstrated in the announcement as comprehensively implemented capabilities; they are conditions to examine in future product evaluation.

Privacy calls for the same caution. The announcement says that video is converted on the device into a sequence of pose coordinates, the original footage is immediately discarded, and the coordinates are sent to a server for translation. Not sending raw video is a step toward data minimization, but pose coordinates derived from bodily movement still raise privacy questions. Evaluation should cover transmission, retention periods, access, reuse, and combination with other data, without equating “not video” with “no risk.”

Enterprise transformation also depends on systems for completion

OpenAI’s RingCentral case study and enterprise usage report describe AI expanding from answer assistance toward execution, while individual experiments become shared workflows. Supporting examples such as PMO status reporting and release governance suggest a hypothesis: value depends not on the model alone, but on a system that includes business context, tool connections, limited permissions, human review, and traceable results.

These materials are also vendor announcements. The reported finding that frontier firms generated 8.3 times as many output tokens as typical firms, Codex’s 64% share of combined enterprise output tokens, and growth in users across job functions are proxies for volume or depth of activity. They do not directly measure productivity, quality, profit, or customer value, nor do they prove that AI adoption caused business outcomes. Other factors, including company size, investment capacity, and willingness to adopt, must also be considered.

An enterprise can instead aim for a repeatable value loop in which AI acts under limited permissions and people feed reviewed results into the next improvement cycle. Effective individual practices become shared procedures, outcomes are verified, and lessons inform the next execution. When that cycle works, experimentation can become organizational capability. Here too, intervention, correction, reversal, and traceable histories should be treated not as announced achievements, but as criteria the editorial team believes should be examined when evaluating adoption.

Using multiple measures instead of one number

The editorial team believes AI’s real-world value should be assessed through multiple measures rather than a single metric. These include coverage across languages, devices, and user groups; task-completion rates; error and correction rates; ease of intervention and reversal; quality differences across people and conditions; review burden; permission violations and incidents; time to completion; sustained use; and eventual value for customers and employees. This is not an evaluation framework the vendors claim to have completed. It is a set of criteria the editorial team considers important for examining possibility and limitation together.

High-performing models remain essential. Durable value also requires co-design with affected communities, natural integration into existing interfaces, minimal data and permissions, traceable decisions, and human review. Without displacing performance, the basis for evaluating AI is expanding toward systems that widen participation and help people complete work safely while preserving a path back from error.