Docs

Testing InfraNistic on Terminal-Bench

Benchmark InfraNistic's cost-optimized routing against single-model baselines using Terminal-Bench's standardized agent evaluation framework.

🔧 Prerequisites

InfraNistic deployed — your AWS Lambda endpoint must be reachable
Terminal-Bench installed — follow the official installation guide from the Terminal-Bench team
Docker installed — Terminal-Bench uses Docker to create isolated environments for each task

⚙️ Integration Approaches

Terminal-Bench offers three ways to integrate a custom agent. For InfraNistic, you have two options:

Option 1: Implement the BaseAgent Interface (Recommended)

Create a custom agent that uses InfraNistic as its underlying model. The BaseAgent interface requires you to implement a run method that calls InfraNistic's API.

Option 2: Use an Existing Agent with Custom Model Config

If you're using an agent Terminal-Bench already supports (Claude Code, OpenHands, Codex CLI), configure it to use InfraNistic as its model provider by pointing to your endpoint.

🚀 Steps to Test

1. Call InfraNistic from Your Agent

InfraNistic exposes a simple HTTP POST endpoint. Wrap it in your agent's model call. For benchmark workloads with unique queries, pass no_cache=true to ensure every call hits the model fresh:

import requests

def call_infranistic(prompt, endpoint):
    r = requests.post(
        endpoint,
        json={"query": prompt, "no_cache": True},
        timeout=600
    )
    return r.json()["response"]

2. Run the Benchmark

tb run \
  --agent your-agent-name \
  --model your-litellm-model-name \
  --dataset-name terminal-bench-core \
  --dataset-version 0.1.1

See available agents and datasets with tb run --help and tb datasets list.

3. Test a Specific Task (Optional)

Filter to a single task for faster iteration:

tb run \
  --agent your-agent-name \
  --model your-litellm-model-name \
  --dataset-name terminal-bench-core \
  --task-filter "specific-task-name"

📊 Interpreting Results

Compare InfraNistic-routed performance against a single expensive model. Focus on:

Metric	What to Look For
Task Resolution Rate	Should match or approach the top-tier model's rate
Cost / Tokens Consumed	Should be significantly lower (target: 3-5× savings)

A successful test: InfraNistic achieves a similar task resolution rate to Sonnet-only, at significantly lower cost. Easy tasks route to Haiku automatically — hard tasks escalate to Sonnet. Net cost is lower than always using Sonnet, with no accuracy regression on tasks within Sonnet's capability range.

🆘 Getting Help

InfraNistic support: support@infranistic.com