kategos
Physical AI

Physical AI: How to Evaluate a Single-Task Pilot | Operations Guide

Evaluate a physical AI pilot around one useful task, with clear measures for quality, interventions, total operating cost, and site-specific safety review.

Physical AI pilot
Physical AI pilot

Give Physical AI One Useful Job

A physical AI investment becomes easier to evaluate when the team can describe its first job in one sentence. Load these parts into this fixture, under these conditions, at an agreed quality level. That gives operations leaders a practical starting point for judging whether a robot improves the work.

The BMW and Figure example is useful when its chronology stays intact. BMW's August 2024 announcement described a several-week trial in which Figure 02 inserted sheet-metal parts into fixtures at Spartanburg. BMW said it was evaluating safe use and integration requirements.

In a separate November 2025 deployment report, Figure reported more than 90,000 parts loaded and 1,250 operating hours, contributing to production of more than 30,000 BMW X3 vehicles. These are supplier-reported results from a later deployment, not figures from the 2024 trial or independent proof of safety.

For another manufacturer, the useful question is what evidence a similarly bounded task would need to produce.

Define the task before selecting the machine

Specify the inputs, acceptable output, physical setting, and conditions outside the pilot's scope. Identify who owns the process and who can stop the trial. Involve operators, maintenance staff, engineering, and qualified safety specialists before deciding where live testing is appropriate.

Compare the proposed robot with credible alternatives. A process change, fixture redesign, conventional automation, or different work allocation may address the same bottleneck. Do not assume AI-enabled equipment is inherently more flexible, cheaper, or suitable for working near people. Those are properties to evaluate in the complete installation.

Measure the whole task

Record a baseline for acceptable units completed, cycle-time variation, defects, and intervention time. During the pilot, use the same definitions and account for setup, maintenance, downtime, and the work transferred to supervisors. Separate scheduled operating time from actual productive runtime so a favorable average does not hide long interruptions.

A useful result should explain the conditions under which the task succeeded and the exceptions that required help. Count a part as complete only when it meets the downstream process's acceptance criteria. Faster movement has little business value if inspection or rework absorbs the saving.

Make the expansion decision explicit

Before the trial, agree on what would justify a larger test, what would require redesign, and what would end the project. Review economics and operational performance alongside the separate, site-specific safety assessment. A strong production score does not replace that assessment.

The next decision may be to expand, narrow the task, or choose a simpler approach. Each can be a useful pilot outcome. The investment case becomes stronger when it explains exactly which job the system can perform, under which conditions, and what the business must provide to keep it working.

BMW Group: Successful test of humanoid robots at BMW Group Plant Spartanburg

Figure: F.02 Contributed to the Production of 30,000 Cars at BMW

Stay Connected With Kategos

Physical AI

Have a problem this kind of work could move?

Tell us what you have. We will make it possible.

Ask kategos Intelligence