Document-Driven Multi-AI Development Process in Naia ADK: Testing Efficiency with Jev

Hello. I'm Luke, creator of Naia.

Naia may look like a consumer product featuring character agents, but a significant portion of my daily work involves software development. Therefore, we build development infrastructure for this and conduct software development for enterprise clients using Naia's development infrastructure. Previously, I published a book titled "Harness Engineering: AI Software Engineering Starting from Re:Zero" (Korean edition, English edition). Since then, I have continued to put significant effort into establishing a better AI agent-based development process.

Today, I am sharing the software development process and artifacts created for Naia's development, along with how we are attempting to introduce Jev, a trending decision model, into this development process.

There are three primary goals I wanted to pursue in this development process: visibility, parallelization, and cost optimization.

• Visibility : Knowing whether development is proceeding properly and, if model drift occurs, identifying exactly at which stage the issue arose.
• Parallelization : Distributing work across multi-agents in parallel to accelerate development speed.
• Cost optimization : Utilizing cost-optimized models. Jev serves as an excellent alternative here.

The basic framework of our harness (operating rules system) is publicly available as open source below.

• Basic framework of personal workspaces and harness (operating rules system): nextain/naia-adk
• Basic framework for team/project collaboration: nextain/naia-pj-adk
• Community participation guide: nextain/naia-comm-public
• Jev is a decision model released by TypeSafe AI.

The task queue, workboard, runner, and planning documents described in this post are still under internal development and remain private. Currently, this process is also in a validation phase, being tested on a new feature for Naia's web platform: the development of Naia Visual Agent Studio, a video avatar capable of lip-syncing and singing. The reason it is not yet public is that it is not yet polished enough for shared team use; we will release it publicly as soon as it is organized.

Abbreviations Used in This Post and Our Development Documents

First, our development documents, issues, and task queues use the following abbreviations as the project's standard vocabulary dictionary. This was introduced because typing lengthy prompts to AI was cumbersome and there was concern about terminology confusion.

1. Background and Problem Recognition

When development is broadly delegated to AI agents, they often start creating user interfaces (UI) without a backend, or report tests run with mock objects as passed. Therefore, we conduct planning top-down and development bottom-up. Planning flows down from the overall user experience, while development builds up from minimal working units, attaching the UI only after the backend has been genuinely penetrated. Starting with screens often leads to massive revisions during integration.

2. Document-Driven Workflow and Development Process

Writing documents first serves to finalize the scope and acceptance criteria beforehand. By documenting requirements rather than issuing simple prompts, we can trace the root cause when problems arise.

We lay out all documents of the development process in a list, and after human verification, create issues and task queue items. Before creating a new issue, the AI examines which documents the issue spans and references previously opened issues. Only when issues, queue items, and test receipts are all properly present can a task be judged as complete.

Development pipeline overview page in the document viewer.

Documents descend in the diagram's order from why we build (PC) down to functional units to build (FE), and architecture plans (PL) are established only after measuring model and engine constraints first. Issues are not split by technology stack layers but kept to one per user value, even when spanning multiple repositories. Processes from backend to validation are designated as checklists within that issue so nothing is missed, and completion is determined only when the entire scope locked by documentation has provided evidence.

Studio integrated index showing issues, implementation locations, and validation statuses across planning document sections in one place.

3. Three-Tier Testing Structure and Sequence Rules

Testing is divided into three tiers following industry standard designations.

• Unit Test (UT): Checks whether a functional unit (FE) operates according to specification.
• Integration Test (IT): Penetrates real backend components end-to-end without UI. Tests that only pass through mock objects are not accepted.
• User Journey End-to-End Test (E2E): Penetrates a single user journey from real browser screens to the real backend. Only units without screens in SP are closed with integration tests without E2E; if screens exist, E2E is required even if the current change is backend-only. The standard is SP, not the implementer's diff.

The key is sequence. Frontend (UI) is developed only after the backend passes integration testing (IT). Currently, this sequence is not mechanically blocked by the harness, but is verified through task contracts and independent reviews via receipts, leaving room for improvement.

Independent validation (QC) runs separately from implementer tests. Without looking at UC and FE, it aggressively verifies whether product promises hold under forced edge-case inputs and exception conditions based solely on PC and SP. Looking at UC and FE would cause reviewers to only check that narrow scope. Since this sits at the later stage of development, we have not yet conducted empirical validation for it.

4. Git-Based Task Queue and Workboard

To ensure trustworthy accountability for who did what and when, tasks are managed through a task queue in a Git repository (naia-comm). There is no shared server yet; the goal is to build a development server after validation, enabling collaboration across multiple devices and developers.

Each participating device clones the repository and pulls periodically to discover new tasks and report task logs. Execution is performed solely by runners registered locally by the device owner (programs that take tasks from the queue and run AI on their behalf); the queue only records the runner's name, not the commands to execute.

Each phase of a task is written as a new JSON file. Execution evidence and exit codes are recorded in result receipts, and cancellations are appended, logging all operations to enhance traceability. The workboard is simply a screen that re-reads and displays these records upon request.

This is the workboard screen (internal URLs are masked). Top metrics aggregate queue records from the naia-comm main branch: at capture time, out of 228 task items, 10 were available, 1 was running, and 65 were currently successful results, with warnings attached to 4 successful records from unregistered runner names.

5. Harness System and Multi-Agent Collaboration Structure

The harness system consists of rules defined by documentation and their verification procedures. Automatic checking mechanisms are currently turned off in recovery mode ("HARNESS OFF" on the board screen), and code-enforced gates do not yet exist, so the coordinator's task contracts, monitoring scripts, and independent reviews enforce the rules.

Using only top-tier models leads to significant cost increases, while using only lightweight models leads to failures in design and validation, derailing the project. Therefore, models are allocated according to task characteristics and cross-validate each other.

※ Model allocations are under testing and subject to change.

Through cost efficiency and privilege separation, high-volume implementation and test iterations are assigned to Gemini 3.8 Flash to conserve top-tier model limits, while suitable models for each role are continuously explored and adjusted. Workers cannot broaden their own permissions, reducing the risk of agents granting themselves privileges and causing problems. However, bugs in this feature frequently cause workers to stall in isolated, unexecutable states, so we are continually testing and improving it.

For example, location checks like "matching declared repository checkout" operate only when passing through the runner, and do not apply to executions launched directly via task contracts.

6. Adversarial Review Based on Independent Investigation

Before opening a submission, the reviewer first directly investigates the original instructions, repositories, commits, and task queue records to formulate their own conclusions, then compares them with the submission. Reviewing only the submission causes reviewers to miss false premises or wrong repositories. A fresh reviewer inspects each round, and the task passes when there are two consecutive rounds without defect findings that change conclusions. If trivial feedback loops repeat, the process halts and escalates to a human for decision.

7. Observed Achievements and Limitations

Observed Achievements

A structure is running where a low-cost model (Gemini 3.8 Flash) implements tasks in non-conversational command-line sessions, the coordinator monitors boundaries via task contracts and monitoring scripts, and top-tier models cross-check after independent investigation in fresh sessions each round. Monitoring scripts display commands executed by workers post-hoc, and reviewers can catch false facts stated by workers.

Observed Limitations and Vulnerabilities

Inexpensive, lower-performance models frequently proceed without following instructions. They report completion with nonexistent queue IDs, insert unrequested exemption clauses into procedural documents, or subtly alter original conditions during summarization. Testing sessions only check whether scripts pass, failing to distinguish whether tests genuinely ran against the real backend.

While independent reviews filter out these defects, verification costs are high because significant effort from top-tier review models is expended on mechanical fact-checking. This is also why we continue testing appropriate model configurations for each role.

8. Verification Efficiency via Jev and Future Tasks

To reduce review burden, we divided validation into three tiers. In the second tier, we are currently conducting technical validation to examine introducing Jev, which offers low cost and high speed.

• First tier, mechanical check (scripts): Things that require simple cross-checking: test receipt pass/fail (0 failures, exit code 0), URL responses, file existence.
• Second tier, type determination (Jev): When integration test (IT) and E2E receipts say "pass", distinguishing whether the test genuinely went through the real backend or merely passed through mocks. Unit tests (UT) originally allow mocks, so they are not subject to this.
• Third tier, directional judgment (top-tier models and humans): Whether scope and intent are aligned.

Jev is a decision model by TypeSafe AI, a low-cost model that responds rapidly using only predetermined choices and probabilities. Software development involves many choice problems, and through continuous measurement, an appropriate threshold can be found to achieve cost and speed efficiency. This optimization approach was widely used in traditional AI software development prior to LLMs, with validation results as follows.

Validation Results

By adopting Jev decisions only when confidence is 0.85 or higher and yields the same answer under rephrased prompts—delegating the rest to large language models (LLMs)—across 871 test files (257 in final evaluation), we measured and estimated that time could be reduced by approximately 66% and cost by approximately 60~70% (time measured against Gemini 3.8 Flash; cost estimated based on unit pricing of models like Opus and Luna).

The cost for 971 Jev calls was $0.22, with Jev taking approximately 0.7 seconds per decision compared to approximately 12 seconds for LLMs.

We are continuing to find optimal values by expanding experiments. While the potential has been confirmed, it has not yet been integrated into the actual development process. Because ground truth only utilized files where both AIs reached the same answer, results may be skewed toward easier files.

Future Tasks

With this procedure, the studio's first feature (entering a script to generate, listen to, and download voice) has been completed from backend through user journey penetration testing. Remaining tasks include further automating validation and having tooling enforce rules currently maintained by humans and contracts. We also plan a separate experiment to test whether Jev can be used not only for validation marking, but also for flow control in choosing the next task when one completes. This flow judgment is an area with significant cost and latency because it currently requires invoking a top-tier model for every task.

I hope the content shared here proves helpful. We would also appreciate your interest in Naia's products. We need to release products quickly to demonstrate outcomes and move to the next stage, yet it feels like we continue spending significant time on the meticulous control and development methodologies of AI.