【Generative AI × Industrial Robots】How Far Can Generative AI Control Industrial Robots?

Chitose Robotics Inc. (headquartered in Bunkyo-ku, Tokyo; President and CEO: Ryoji Nishida; hereinafter "the Company") has conducted a verification study on the impact of reference information provided to Vision-Language Models (VLMs) on the quality of industrial robot control program generation, using a VLM-based robot operation instruction system.

In this verification, the system under evaluation enables VLM coding agents (Codex, Copilot, Claude Code) to generate C++ control programs for industrial robots based on hand-camera images and Japanese-language work instructions. The study progressively added three types of reference information to the VLM: an "embedded prompt" describing prior knowledge specific to industrial robots, an "API reference" for robot, camera, and sensor control, and a "past case database" containing real-world operational data, evaluating the effect of each on program generation quality.

The results confirmed improvements in task instruction adherence, safety considerations, implementation using actual equipment control APIs, and code readability and maintainability when reference information was organized and provided by role. Notably, the past case database showed potential as a source enabling VLMs to access field-derived tacit knowledge embedded in practical code.

Related to this verification, a video demonstrating the actual operation of the VLM instruction system will be published on YouTube.

The video showcases an industrial robot performing pick-and-place operations based on camera images and Japanese-language instructions. Watching it alongside the research findings allows for a more concrete understanding of the potential of VLM-based robot operation instruction.

Background

The use of industrial robots in manufacturing is rapidly expanding. There is growing demand for systems that allow on-site workers to directly instruct robots using generative AI. However, knowledge about how to continuously improve the quality of robot operations has been limited, and clear directions have not yet been established.

This verification aims to clarify what is needed for a system to generate code that meets specifications and ensures operational safety in response to instructions from general on-site users without robot teaching expertise, while also improving code proficiency comparable to that of experienced developers.

Verification Scope

This verification focused on standalone pick-and-place tasks.

Operation instructions included object selection, such as "grasp the red workpiece detected by the camera and place it at a designated position," "grasp the blue workpiece and place it at another designated position," and "grasp the leftmost workpiece and place it at the center of the tray."

Safety-related conditions were also evaluated, including "if no workpiece is found, do not move the robot and terminate with an error" and "after grasping, always retract 50mm in the Z direction before moving."

The physical system used an industrial robot equipped with a hand-eye camera, laser sensor, and small pneumatic chuck hand.

Verification Method

Human operators provided work instructions in Japanese, and VLM coding agents interpreted these to automatically generate control programs for the industrial robot. The reference information provided to the AI was categorized into three types, and the impact on "compliance with instruction specifications" and practical "code proficiency" was evaluated.

Embedded Prompt: Rules describing fundamental structures and prior knowledge specific to industrial robots, such as avoidance maneuvers for collision prevention.

Guidebook and API Reference: Foundational knowledge and detailed specifications for system construction, including control of robots and cameras.

Past Case Database: Real-world practical code examples, including past project files used on-site and safety designs for error handling.

Verification Results

The verification used an industrial robot equipped with a hand-eye camera and laser sensor, targeting a total of 12 tasks, including pick-and-place operations on differently colored parts (workpieces). Human operators provided instructions in Japanese, and the VLM automatically generated control programs.

The programs generated by AI were scored based on two criteria:

Compliance with Instruction Specifications: Evaluation across 10 items (maximum 20 points per task), including whether the task objective was achieved, correct image recognition of target objects, and inclusion of safe collision-avoidance movements and exception error handling.

Code Proficiency: Assessment on a five-point scale (maximum 5 points per task) of whether the code featured a practical, readable design—properly modularized and easy to maintain and modify by humans—rather than merely functional code.

The results showed that as reference information was progressively added to the AI, the overall score increased significantly from 74.3% to 88.7% of the maximum possible score (+14.3 points).

Reference Information Provided to AI

Specification Compliance

Code Proficiency

Total Score

Standard Sample Only

191 points

32 points

223 points

+ Embedded Prompt

202 points

39 points

241 points

+ Embedded Prompt + API Reference

216 points

40 points

256 points

+ Embedded Prompt + API Reference + Past Case Database

222 points

44 points

266 points

Furthermore, distinct roles were identified for each type of reference information provided to the VLM.

Embedded Prompt

Role: Supplements fundamental implementation practices for industrial robot systems.

By adding basic rules such as "always retract upward after grasping," the AI gained an understanding of the conventional practices unique to industrial robot systems, improving both compliance with specifications and basic code proficiency.

API Reference

Role: Aligns VLM outputs with real APIs and correct implementation procedures.

While generative AI can produce natural-looking code, it must avoid using non-existent APIs or incorrect calling sequences in real-world control. Providing detailed specifications of actual functions constrained the AI's output to correct implementation procedures, further enhancing responsiveness to human instructions.

Past Case Database

Role: Supplements tacit knowledge embedded in practical code from real-world operations.

Providing field-specific practical code not documented in manuals enabled the AI to learn "tacit knowledge" such as safety designs for error handling, resulting in the highest quality generated code. This past case database is expected to serve as a powerful differentiating factor in industrial AI systems.

Discussion

This verification suggests that for generative AI-based industrial robot control, it is essential not only to instruct the model on task content but also to properly organize and provide references on robot control conventions, API specifications, and practical code.

In industrial robot environments, tacit knowledge accumulated by experienced engineers through practical work is crucial, beyond just documented manual knowledge. Implementation for safe retraction, error handling, maintainable code structures, and task segmentation based on equipment configuration are areas difficult to fully reproduce with simple sample code alone.

Going forward, it will be important to structure and prepare internal implementation knowledge and operational know-how in a format accessible to AI for effective use of generative AI in robot control.

Future Outlook

Future efforts will include continuous operational testing in real-world settings aligned with on-site needs, application to complex procedural tasks such as assembly and fitting, improvement of algorithms for searching and selecting reference files, and development of templates to suppress noise when referencing past code.

Chitose Robotics aims to combine generative AI and industrial robot technologies to create systems that make robot utilization easier not only for specialized engineers but also for on-site workers.

Related Presentation

This work will be presented at the Robotics and Mechatronics Conference 2026.

Event Name: Robotics and Mechatronics Conference 2026

Venue: Fukuoka International Congress Center (2-1 Ishioka-machi, Hakata-ku, Fukuoka City, Fukuoka Prefecture 812-0032)

Presentation Date and Time: June 30, 2026 (Tue) 14:00–15:30

Presentation Title: Robotization in Manufacturing 1P1-008

"The Role of Embedded Prompts and Databases in a VLM-Based Industrial Robot Operation Instruction System"

Presenter: Ryoji Nishida (President of the Company)

Registration: https://robomech.org/2026/registration/

Chitose Robotics Inc.

Our YouTube channel features numerous robot demonstration videos.

Please visit to see the latest robots and robot hands in action.

YouTube Channel

This is the link to our YouTube channel.

Chitose Robotics Inc. aims to realize a future where robots are utilized across more workplaces and industries through the social implementation and dissemination of robotics technology.

We will continue striving as a bridge between technology and real-world applications, working toward a society where robots are more accessible, support human work, and create new value.

Head Office (until June 30): 2-1-1, Koshigaya, Bunkyo-ku, Tokyo 112-0002, 7th Floor

Head Office (from July 1 onward): 1-5-7, Hongō, Bunkyo-ku, Tokyo 113-0033, 2nd Floor

President and CEO: Ryoji Nishida

Phone: TEL 03-5615-8271 / FAX 03-5615-8272 Business Activities: Development and sales of robot control software "Kurubo" Research, development, and sales of robot systems URL: https://chitose-robotics.com/