Can AI "Read the Room"? InfiniMind Releases "NARU Bench," an Evaluation Platform for Japanese Video AI

Multimodal models have made remarkable progress in "watching" videos. They can now generate captions for short clips, answer questions about what's on screen, and even search for specific moments within hours of footage. However, asking a model to track the shifting motivations of a person in a two-hour interview or to interpret the subtle meaning behind a Japanese "aizuchi" (interjection) suddenly raises the difficulty level.

Today, we are releasing "NARU Bench (Narrative And cultural Reasoning Understanding)," which tackles precisely this challenge. It is an open benchmark built to measure the "video comprehension" truly essential in real Japanese media and to reveal the gap that currently exists before models can reach human-level understanding. This benchmark was developed through a joint research project between InfiniMind and the University of Tokyo.

NARU Bench is currently available on Hugging Face and GitHub and can be run immediately using lmms-eval.

Why This Benchmark is Necessary

The Japanese video content market reached approximately 630 billion yen (about 4 billion US dollars) in 2025. This type of content is not only massive in scale but also extremely high-context. Much of the meaning is conveyed through the atmosphere, indirect expressions, and interpersonal dynamics, rather than what is explicitly stated in speech. Evaluating whether AI systems truly understand such content requires an approach that goes far beyond simple object recognition or clip-level Q&A.

While existing video benchmarks have advanced the field, they share common blind spots:

Bias towards English: The majority of existing benchmarks use English content, with little validation of non-English media or the "cultural reasoning ability" required for it.

Focus on Short Clips and Search: Even benchmarks for long-form videos often only measure a model's ability to find or recall explicit information, without questioning its capacity to maintain a consistent narrative model over time.

Low-Context Assumption: They rely on media where important information is stated plainly, thus failing to measure understanding of implicit social cues and culturally rooted meanings.

NARU Bench sits at the intersection of the two most challenging of these issues: long-form narrative tracking and high-context cultural reasoning, evaluating them using actual Japanese videos.

Overview of NARU Bench

This benchmark consists of 1,481 multiple-choice questions created from 155 Japanese YouTube videos. The content spans 30 to 240 minutes, totaling approximately 147 hours of interviews, discussions, and talk shows. All questions and choices are written in Japanese, and require actually watching the video to answer correctly. These questions are designed to be unanswerable with only the transcribed text or linguistic prior knowledge.

The benchmark spans two tracks and nine task types:

Narrative Comprehension (745 questions)

Cultural Comprehension (736 questions)

Dataset Construction Methodology

Details of the construction methodology will be described in an upcoming paper, but the key points are as follows:

Source Data: From a pool of approximately 100,000 videos, we selected Japanese-centric content using Automatic Speech Recognition (ASR) and filtered by playback duration (30 minutes or more) to narrow it down to about 8,000 candidates. Reviewers then curated the final set based on video completeness, narrative continuity, social interaction, and semantic diversity.

Annotation and QA Generation: We built an agent-based annotation platform that maintains narrative continuity between segments using a hierarchical memory mechanism, while dividing long videos into chunks. Structured annotations (existence, events, plot, cultural cues) are fed into a constraint-based question generator. An iterative debiasing loop includes a blind solver that attempts to answer each question without watching the video, detecting and correcting items that can be solved with text alone.

Human Verification: For all questions, native Japanese speakers verified accuracy, relevance, difficulty, and whether true video comprehension was genuinely required.

Evaluation Results: The Wall Faced by Existing Models

We evaluated multiple models on the 1,481-question benchmark.

The results are clear. With four choices, random guessing yields a 25% accuracy rate—and some of the leading open-weight models barely exceed that level. Even the most powerful general-purpose open-weight model we tested reached only 33.9%. Our DeepFrame 8B achieved 32.1%, comparable to much larger models like the 30B Qwen3-Omni. What's truly noteworthy, however, are the platform-level results. DeepFrame Platform, which combines retrieval and agentic reasoning across the entire video, achieved the highest score among all systems tested at 55.3%. This is about 1.6 times better than the best standalone model. Even so, a significant gap remains to reliable human-level understanding. NARU Bench is truly difficult, and that's precisely its significance. It exposes failures in narrative comprehension and cultural reasoning that never surface in general video QA.

Try It Now

NARU Bench is directly integrated with lmms-eval and can be easily run without forking.

The dataset is available at infinimind/naru_benchmark. You can run all tasks together or evaluate individual subtasks (e.g., narubench-cultural-2.1.aizuchi). Source videos are referenced by ID. Researchers requiring an archived copy can request access via the link in the dataset card.

About the Joint Development

NARU Bench was jointly developed by InfiniMind and the University of Tokyo. It was built in the context of GENIAC (Generative AI Accelerator Challenge), a government-led initiative to strengthen AI foundation model development in Japan. NARU Bench provides an evaluation platform for understanding Japanese media, an area inaccessible to English-centric benchmarks.

Future Prospects

Expansion of Source Data—Beyond YouTube, to Japanese television and structured broadcast media

Methodology Improvement—Continuous refinement of automated pipelines and human verification

Paper Publication—Preparing a paper summarizing construction, methodology, and comprehensive evaluation

NARU Bench is available starting today. We hope this benchmark will be useful to everyone building and evaluating multimodal models that need to handle real Japanese videos—models that must not only recognize what's on screen but also understand its meaning.

📦 Dataset: huggingface.co/datasets/infinimind/naru_benchmark

💻 Code: github.com/infinimind-inc/naru_benchmark

The dataset (questions, choices, metadata) is released under CC BY-NC-SA 4.0. The evaluation code is released under Apache-2.0. Referenced videos are subject to the terms of use of their original creators and YouTube.

Company Profile

Company Name: InfiniMind Inc. (Japan Branch: InfiniMind K.K. (formerly) SDio Inc.)

Representative: Kai-Uwe (Representative Director)

Established: November 2024

Location: 214 Homer Ave, Palo Alto, CA 94301 USA (Japan Branch: Otemachi, Chiyoda-ku, Tokyo)

Business Activities: Development and provision of large-scale video foundation models and AI search platforms for enterprises.

URL: https://infinimind.io

About InfiniMind

InfiniMind is building AI infrastructure specialized in video that transforms the vast amount of video data existing worldwide into "structured data" usable for business. Led by founding members with experience leading video recommendation research and data science teams at Google, the company is engaged in R&D of cutting-edge Large Video Models (LVMs). Since its full launch in November 2024, InfiniMind has been at the forefront of Deep Tech, aiming to establish the global standard for video foundation models. It has been selected for the 3rd phase of GENIAC (Generative AI Accelerator Challenge), a government-led project to support generative AI development, for major platform startup programs by Google, NVIDIA, and others, and as one of only two companies in Japan selected for AWS GAIA 2025 (Generative AI Accelerator). The company is challenging the transformation of the global market from its base in Japan.

We are actively recruiting colleagues to join us in competing globally. Please inquire via Linkedin or the email below.

Contact for Inquiries Regarding This Matter

Email: [email protected]