Building a distributed platform for agent training and evaluation

Updated 21 Oct 2026

At SynthLabs, I designed and built a distributed platform for running, evaluating and training tool-using agents. Its execution and learning core powered campaigns involving tens of thousands of agent attempts. The platform runs isolated repository tasks, captures the model interactions produced inside the native coding harness, joins that experience to task assessments, and trains a supported model from it. The resulting checkpoint returns to the same harness.

The central engineering problem was keeping native execution, learning evidence and model state in agreement. The harness had to retain its tools, delegation and context behavior; learning had to use the exact tokens sampled by a particular model snapshot; and each update had to yield usable state for subsequent work. Independent task execution and learning preparation could overlap, while policy advancement retained an explicit synchronization boundary.

I built the Python RL Runtime, native model adapters, reusable agent profiles, and the managed delivery paths around them: a Go CLI and agent launcher, authenticated control, storage contracts and cloud infrastructure. An operator can submit versioned work and an agent recipe, then return to its execution, assessments, artifacts and checkpoints without operating the underlying workers.

The experiment is the interface

An agent experiment includes the work environment, executable agent, assessment and policy for running it. A coding agent may inspect files, edit code, run tests and react to their output. Its result includes both the changed artifacts and the interactions that produced them.

The platform gives those inputs distinct homes:

A task defines a piece of work. For a repository-level engineering task, that might be a pinned checkout, a problem statement, installed dependencies and tests. Synth uses Harbor’s task format: the instruction, environment and verification travel together. A dataset selects exact task versions in a defined order.

The profile defines the agent. The same task can be attempted with different models, instructions, skills or tool configurations. Those choices belong to the agent rather than being baked into every task.

The assessment defines what counts as useful work. A task can provide programmatic tests. Where the task supports a separate verifier environment, an explicit judge profile can instead assess an output using its own model, instructions and reference information. Actor and judge are roles using the same underlying agent abstraction.

The plan defines how the experiment runs. Evaluation can repeat a fixed setup across a dataset. Training chooses batches of tasks and repeated attempts at each task, with explicit optimization and execution settings. Unscored execution uses the same task-performing path when the desired result is simply the work and its artifacts.

An engineer can evaluate a baseline profile, change its instructions or tools, and compare the new setup on the same versioned work. Supported training then updates the model while retaining the chosen recipe; the checkpoint can be evaluated on held-out tasks through that same harness.

An authorized operator works with commands such as:

synth run start \
  --dataset team/evaluation@v1 \
  --actor @agent.json \
  --plan @eval-plan.json
synth train start \
  --dataset team/practice@v1 \
  --actor @agent.json \
  --plan @train-plan.json
synth run show <run-id>
synth run events <run-id> --follow
synth run checkpoints <training-run-id>

The dataset, profile and plan references above are illustrative. Submission returns a durable run identity; the operator can return to its progress, results, artifacts and checkpoints without keeping a local controller running.

The system underneath

The architecture separates the control plane, which authorizes and records the experiment, from the execution and model-computation paths, which do the work. Large task packages and trajectories move through object storage rather than passing through the control service on every transfer.

The main components have deliberately different jobs:

• synth is the operator interface and native-agent launcher. It exposes reusable work and agents, submits experiments, and reads their results.
• sprite-gateway is the authenticated control plane. It resolves authorized inputs, admits durable jobs, delegates execution and serves run progress and results.
• agent-rl-runtime schedules task attempts, coordinates evaluation and learning, manages progress and advances model state.
• managed-agent-proxy connects the native harness to supported models and preserves the model-interaction evidence needed for learning.
• PostgreSQL and object storage retain complementary parts of the system: authoritative relationships and lifecycle facts in one; task packages, trajectories and other large payloads in the other.
• Kubernetes, Harbor, sandbox providers and model-computation providers supply the service, task-execution and numerical-computation foundations.

Consider one training batch. Runtime resolves its selected tasks and agent configuration, starts isolated attempts, and gives each native agent narrowly scoped model access. The agent’s tool use occurs in its task environment; its model calls pass through the compatibility and capture layer. A verifier or judge assesses the completed work. Runtime joins those assessments to the corresponding model interactions, prepares learning inputs and submits them to the training backend. After the update, it obtains the sampler used by the next batch and retains a checkpoint that an authorized operator can inspect and use.

An agent profile is an executable recipe

Agent behavior changes faster than execution infrastructure. A different instruction set, skill or supported harness configuration should be a new experiment input, not require a new Runtime deployment.

I developed the agent-profile abstraction to make that complete setup reusable outside the execution machinery. A profile records a selected model and an executable recipe: the native engine and version, supplied files, setup, working directory, command arguments, environment bindings, instruction delivery and required outputs. Instructions, skills, hooks and tool configuration can be supplied through that recipe.

This creates two important experimental levers. To investigate a harness change, keep the model fixed and change the agent’s instructions, tools or context. To investigate a model update, keep the recipe fixed and bind it to a new checkpoint. The system can retain both identities without treating a saved model as the entire agent.

The local profile runner, synth agent run, and Harbor-managed execution use one Go interpreter for the profile. Runtime supplies task, run and effective-model bindings to that launcher; it does not reconstruct the recipe in Python. Instruction delivery, literal arguments, setup, environment and working-directory resolution, and required outputs therefore have one execution contract.

Required routing, capture and model-compatibility settings are checked as launch constraints; conflicts are rejected. Other native settings retain the harness’s configuration hierarchy rather than being silently replaced by runner defaults.

Published profile versions are immutable, so an experiment retains the version it selected when a newer one is published. The service supplies canonical profile bytes and a digest; the Go client verifies that representation instead of assuming that Go and Python serialize the same object identically.

During training, each batch binds that fixed recipe to an explicit effective model snapshot. The experiment can advance the policy without silently changing the harness.

Actors and judges can run independently

An actor profile defines how to perform work. A judge profile defines how to assess its artifacts. Both are complete agent recipes, with independently selected models, instructions, tools and execution configuration. A reusable judge can serve a family of tasks while each task supplies its own reference material. A strong judge need not be a model the training backend can optimize. On eligible tasks, it runs in a separate verifier environment with task-owned assessment context, declared actor outputs and its own scoped model access.

The roles support three useful modes. Actor-only execution produces work and artifacts without requiring a score. Evaluation pairs fresh attempts with the task’s verifier or an explicit judge. Reassessment applies a new or repeated judge to retained work without asking the original actor to solve the task again.

I built synth run regrade by composing the task registry and evaluation machinery. It derives tasks from the original immutable task versions, restores retained actor outputs, and runs the selected judge against the preserved assessment context. Source-trial and artifact digests record which work was assessed. Different judges and repeated assessments can therefore examine the same retained output; the task and judge still define what that output means.

This makes the evaluator an independent experimental choice. A judge can be calibrated or checked for consistency while holding the original work fixed. Weight training remains a separate operation: it additionally requires the original actor’s captured sampled-token evidence.

Keep the native harness, change the model

I wanted to retain the capabilities of a real coding harness: repository navigation, shell commands, file editing, tool use, native configuration and configured delegation to subagents. Synth’s model layer lets the installed harness use a supported backend model while preserving that execution model. In the managed native path, delegated agents inherit the selected model and scoped inference route rather than silently switching to an unrelated policy.

The same integration reaches a developer’s own workspace through synth claude. It launches the installed Claude Code with a supported base model or an owned checkpoint selected with --model, retaining native tools, plugins, permissions and resume behavior. The launcher supplies compatible model settings; Gateway authorizes the selected model, and Proxy routes inference and captures its interactions. This local interactive entry point is distinct from running a saved agent profile or submitting a managed experiment.

Here, Claude Code is the native harness and speaks Anthropic’s Messages protocol. The models behind Synth may have different tokenizers, chat templates, tool-call grammars, reasoning formats and response framing. Supporting them requires a model-aware compatibility layer.

The adapter renders the conversation and tool definitions using the model’s native format, tokenizes that input, submits generation, and parses the result back into the response the harness expects. The harness then performs the tool action and continues the conversation.

Compatibility also includes the selected model’s context and output limits, and model-native token counting, so the harness can manage context against the backend it is actually using.

For example, a model’s native request to run pytest must become the structured tool-use response Claude Code expects. The harness executes the command in the repository environment. Its output then becomes an observation in the next model-native prompt. The learning record retains the actual generated tokens and their probabilities alongside that tool cycle.

I built model-specific routing and adaptation across families including Qwen, DeepSeek, Kimi and GLM, with backend selection and native rendering/parsing kept behind explicit adapter interfaces. The inference layer and training admission have separate capability checks: a route can be useful for inference without being a model that the selected training backend can optimize.

For training, the Proxy retains the model-native record of each call: prompt token IDs, generated token IDs and the generation policy’s log probabilities. Those values come from the sampling path itself and are joined to their interaction and task attempt. The native response and learning record come from the same prepared prompt and sampled output; reconstructing a usable tool response does not replace the original tokens with a new tokenization.

A displayed transcript is not a complete substitute. Chat templates, tool syntax, reasoning and tokenization affect what the model actually saw and generated. The harness can compact earlier conversation; separately, the Proxy can fit oversized tool results and expanded skill content to the selected model’s prompt budget, using explicit omission markers. Per-call capture preserves the effective input actually submitted for sampling, so learning follows the contexts used rather than assuming that every prompt contains the full displayed history.

Separate serving from evidence publication

Model adaptation also has its own scaling constraints. Rendering long prompts, tokenizing them and parsing model output are CPU work; they should not block the HTTP event loop that manages concurrent requests. I separated that work into a dedicated worker pool, with independent admission for generation and token-count requests.

For the Qwen path, I also implemented exact-token prefix reuse across growing conversations. A bounded cache is scoped by owner, credential, model, native session and delegated-agent identity. It reuses a completed-turn prefix only when its rendered text matches exactly, then tokenizes the new suffix. This reduces repeated local tokenization without changing the prompt supplied to the model.

Evidence has a separate lifecycle. For captured calls, the service stages the interaction before returning the completed response. Response-delivery settlement hands the staged record to a bounded publisher, which obtains an upload grant, transfers the payload and commits its reference. Foreground model preparation, staging and publication have separate execution resources. A slow bulk transfer does not have to occupy the agent’s response path. Response delivery, published byte integrity and eligibility for learning are distinct checks.

One execution core for evaluation and training

I organized execution, evaluation and training around one common unit: one task, a fixed agent setup and a requested number of attempts. Runtime calls this a sampling group. Its attempts share an effective model snapshot. In the profile-based path, the accepted run also pins the executable harness recipe.

Multiple groups can execute concurrently. In the profile-based path, a Runtime worker coordinates an accepted run and fans its independent attempts out to isolated remote task environments. Each attempt has a stable correlation identity connecting its execution, assessment, artifacts and model interactions. The runner handles the Harbor/environment lifecycle; the agent performs the work inside it.

Model inference and training computation run outside the task environments, through backend services. The worker coordinates state and results across those execution and model-computation paths.

This core supports three uses:

• Unscored execution returns the work and artifacts.
• Evaluation aggregates the outcomes and assessments of a fixed setup.
• Training adds the evidence preparation and model-update path to those same task attempts.

Evaluation does not need to initialize an optimizer or construct training datums. Training does not need a second task-execution architecture. The two operations share sampling and diverge at the point where their purposes differ.

Scale follows resources and independent work

A plan sets total work; resource admission controls what can run at once. Runtime tracks active attempts, CPU, memory and storage against the execution budget, using task metadata and supported overrides for sandbox shape and phase deadlines. Heterogeneous tasks can execute concurrently within those limits.

For example, several memory-heavy repository builds can fill the RAM budget while trial slots remain available. Further admission waits for the resources it needs. The model service can meanwhile have a different bottleneck, and trajectory export can use its own concurrency limit. One global semaphore would not describe those constraints well.

Runtime reads the selected immutable dataset version in pages. The control request carries its identity and authority; the client does not have to expand a large dataset into a giant submission. Task materialization and preparation happen as part of the execution path.

From the operator’s perspective, execution is serverless: submit an experiment and consume its outputs while the platform manages workers, task environments, model access and lifecycle. Workload size, active task resources and model-computation throughput remain separate controls.

From experience to learning

Training needs both the task outcome and the native model interactions that produced it. A task assessment belongs to a complete attempt; the individual generating calls supply the token-level learning inputs. Each attempt can contain many model calls and tool interactions.

In the group-relative setup, Runtime samples several attempts at the same task from the same model snapshot, with the harness recipe held fixed. Their assessed outcomes provide a relative learning signal for that task.

The implemented path is GRPO-style LoRA training through Tinker. Runtime joins each assessment to its originating attempt and model records, validates finite rewards and checks output-token / log-probability alignment. The learning inputs use the actual prompt tokens, sampled output tokens and generation-policy log probabilities.

Eligibility is settled before relative advantages are computed. A valid zero reward is a learning outcome; absent or invalid trainable evidence is a different state. Runtime qualifies the retained trajectories, applies the group’s eligibility and signal requirements, and records exclusions. Tinker Cookbook then centers rewards within the qualified task group to produce relative advantages and construct the learning datums. An attempt must not change its siblings’ advantages and then disappear because its evidence could not be used.

Within an attempt, generated action tokens receive its group-relative advantage. Observations provide context and receive zero advantage: the optimizer is not being taught to generate the shell’s output as though the agent wrote it.

Captured calls from the actor’s delegated agents follow the same rule: their sampled outputs are actions in their own contexts. A child’s answer, returned as a tool result, is context for its parent. These calls share the attempt’s group-relative advantage.

Per-call capture does not require one training datum per call. Cookbook packs successive interactions while the next prompt extends the accumulated prompt and generation. Earlier generated actions keep their weights; newly appended observations receive zero advantage. When that exact prefix relationship breaks, packing starts another datum with the next call’s actual context. A multi-turn trajectory can therefore yield several datums without imposing a fictional append-only conversation on the harness. The default trainer passes the captured generation-policy log probabilities and these advantages to Tinker’s importance-sampling loss.

Harbor supplies task/trial execution. Tinker and its Cookbook supply model computation and learning primitives. I built the native-agent execution and model adapters, the joins from task outcomes to token-level learning inputs, and the coordination that advances usable model state.

Overlap the work, preserve the policy boundary

A task group being ready for learning and the policy being ready to advance are different boundaries. Agent attempts have uneven runtimes, so I implemented a grouped pipeline that lets a ready group export evidence, prepare datums and submit forward/backward work while other groups in the same batch are still sampling.

Sampling and training use distinct state interfaces. Each group generates through the batch’s captured sampler, S₀, while the trainer accumulates learning contributions. A ready group can submit its work as slower groups continue sampling through S₀.

When a batch has eligible learning contributions, the coordinator waits for its groups to settle and their contributions to be submitted, then enqueues the optimizer step in the backend’s ordered operation stream. It settles that work, saves the next sampling snapshot, S₁, and binds the following batch to it. The harness recipe stays fixed across that model transition.

This overlaps independent work while preserving the policy boundary. Every attempt in the current batch keeps its captured sampler, S₀; the next batch uses S₁.

Checkpoints are usable model state

Training produces more than a scalar metric. Runtime coordinates saved inference weights, the sampler used for subsequent generation, and the trainer state needed for supported optimization continuation. Its resumable state also tracks the dataset position and acknowledged training progress.

An operator can inspect a checkpoint, bind its saved model to the same harness recipe and evaluate it on another task set. Starting new training from that model and continuing compatible optimizer state are distinct choices. The checkpoint’s ownership and training context remain associated with its identity. Inspection links it to the producing run, later runs that selected it, and sampling groups recorded against its saved model state.

Resumability also includes the boundary around a remote optimizer step. Before enqueueing it, the lifecycle durably marks the update as unresolved; acknowledged progress is recorded separately. An unresolved update blocks automatic continuation: a missing acknowledgement is not treated as permission to repeat the update or advance the dataset cursor. A saved checkpoint and an outstanding remote mutation are distinct pieces of state.

This closes the practical loop: task experience can produce a new model, and that model can return to the same native agent for further work or evaluation.

A managed service around the learning loop

The Runtime is only part of the platform. Turning it into a remotely operable, shared service requires durable admission, bounded execution authority, authorized storage and a useful inspection surface.

Durable jobs and scoped execution

Gateway resolves the dataset, actor and optional judge, checks their authority, and pins the accepted specification. I made acceptance one database transaction containing the run, immutable specification and authoritative creation event. An exact retry with the same idempotency key returns that accepted identity. Workers can claim the job only after its execution prerequisites are durably ready. Organization-level admission governs jobs; Runtime admits the actual task resources within each job.

For profile-based jobs, claim admission checks the required profile and optional-judge contracts against advertised worker support, and separately checks the authorized sandbox provider. Runtime independently validates the pinned job before downloading task inputs or creating a sandbox.

The customer identity, worker service identity and authority for an individual attempt are distinct. Workers claim accepted work and receive scoped execution credentials. Per-trial model access is bound to the run, attempt, owner and selected model; the sandbox receives scoped model access rather than a shared provider key. Heartbeats maintain ownership and convey cancellation.

At the managed-run boundary, PostgreSQL checks the worker’s active execution attempt and identity inside the transaction that appends progress or commits completion. Progress retries are deduplicated by that attempt’s local sequence. Completion locks the run and its current worker attempt, checks the final event boundary and transitions both together. Ownership therefore governs the actual state changes, not just the initial API request.

Metadata and bulk data take different paths

Training trajectories and task archives can be large. Sending every payload through the authorization service would tie data throughput to the control plane’s memory, bandwidth and request handling.

For task materialization and training evidence, I separated those paths. Gateway authorizes transfers and commits canonical references. Producers and consumers transfer payload bytes directly to and from object storage. PostgreSQL retains the relationships, lifecycle facts and object references needed to inspect and authorize them.

I designed the canonical interaction-storage contract around exact, create-only uploads and provider-verifiable full-object SHA-256 checksums. Uploading an object, publishing its verified reference and authorizing a read are distinct operations. Components can share the same content reference without copying evidence merely to change which service refers to it; each consumer still needs its own read authority.

Cloud delivery, networking and credentials

I built the native-agent inference service’s cloud delivery boundary on the team’s Kubernetes/Azure substrate. Public ingress exposes native model traffic and readiness; service control uses a separate authenticated path. Application releases use pinned images and Helm configuration, while infrastructure owns the network, durable staging and credential-delivery substrate.

Dedicated workload identities read the required credential objects. A secret controller materializes exact versions into credential generations that application releases consume by reference. This keeps secret values out of agent recipes, Terraform plans and Helm configuration, while preserving the selected credential generation for release and rollback. Worker lifecycle handling coordinates claimed work, heartbeats, cancellation and shutdown.

The key separation is between the agent’s permissions inside its work environment and the platform’s authority to operate infrastructure. A task-performing agent can be equipped with powerful tools in its sandbox without receiving the operator’s database, cluster or model-provider administration credentials.

Inspection is part of the interface

Long-running work should be observable from another terminal or machine. Gateway retains ordered events and result references so inspection does not depend on finding a particular worker process. Clients can reconnect to an event stream from a cursor and return to the same run identity.

I designed inspection around experimental lineage: which frozen agent produced each attempt, which trials and token spans supplied a training group’s signal, and which groups contributed to a model update. The retained records connect that evidence to the successor model snapshot and checkpoint. Large token payloads stay in artifacts; bounded evidence chunks preserve the exact references without forcing a whole trajectory into a control-plane event.

The Go CLI exposes human-readable views and structured JSON for automation. Status, events, artifacts, task attempts, learning updates and checkpoints are separate useful views of the experiment. Bulky outputs remain downloadable artifacts rather than being forced into terminal messages.

This matters both for a researcher monitoring training and for another agent operating the platform. A program can submit work, retain its handle, inspect the outputs and decide what to do next through the same interface a human uses.

What this made possible

I used the platform for training and evaluation campaigns involving tens of thousands of agent attempts, repeated model updates and checkpoint continuation. Actual workloads included R2E-Gym repository tasks from public Python projects such as pandas, NumPy and aiohttp, with task-specific tests and pinned environments. Other task collections used the same execution and learning machinery: the task package defines the work and assessment.

The execution and learning core supported those campaigns. I subsequently generalized agent behavior into reusable profiles and exposed the machinery through the unified managed interface.

The system was deployed and exercised internally in the development environment; it is not generally available. The public documentation and CLI releases provide inspectable interfaces to the work.

Profiles gave task-specific actors and judges a common execution format. I used these boundaries in subsequent work on task factories and repository-specific reviewer construction: an agent-producing workflow can return a runnable recipe that a task-performing workflow consumes. The companion article, From real software work to specialized agents, explains the task production, knowledge construction and applied workflows.

My contribution was the design and implementation across these boundaries: native-agent/model integration, distributed task and learning orchestration, reusable profiles, durable data contracts, managed control, and the CLI and cloud systems that expose them as one capability. I built this with the SynthLabs team and extensive use of coding agents, adopting existing execution and model-computation foundations where they fit.

Related articles

• Model specialization and training agents — the earlier distillation, training-agent and data-factory work.
• From real software work to specialized agents