---
title: Run an Evaluation Suite
description: Execute a Vector Configuration in a disposable Fixture and inspect normalized response and effect evidence with vitest-evals.
sidebar:
  label: Run evals
---

`createOpenCodeHarness()` adapts one OpenCode prompt to the `HarnessRun` contract from `vitest-evals@0.16.1`. Vitest-evals owns Evaluation Suite cases, judges, execution, and reporting.

Use Node.js 24 on macOS or Linux. Install `vitest-evals` at `0.16.1` and provide a compatible optional `@opencode-ai/cli` peer. Commit the Fixture and OpenCode Plugin Entrypoint used by the suite. Treat them and every command they can run as trusted code.

Create the evaluation around your committed Fixture and entrypoint.

```ts title="tests/configuration.eval.ts"
import { createOpenCodeHarness } from '@evilkiwi/vector/evals';
import { expect } from 'vitest';
import { describeEval } from 'vitest-evals';

const harness = createOpenCodeHarness({
  fixture: new URL('./fixtures/repository/', import.meta.url),
  plugin: new URL('./fixtures/opencode.ts', import.meta.url),
  agent: 'evaluation-agent',
  model: { providerID: 'provider', id: 'model' },
  environment: {},
});

describeEval('Vector Configuration behavior', { harness }, (it) => {
  it('returns the expected result', async ({ run }) => {
    const result = await run('Complete the Fixture task.');

    expect(result.output).toContain('complete');
  }, 35_000);
});
```

Run the Evaluation Suite through the repository script that owns your eval files. In Vector's repository, the command is:

```sh title="Terminal"
pnpm test:evals
```

A passing case produces a normal Vitest pass and a `HarnessRun` available to its assertions and reporters. Each `run()` copies the Fixture into a disposable temporary repository, starts a fresh supported `opencode2 serve` process, observes the session through terminal completion, collects evidence, stops the process, and removes temporary state.

## Inspect the evidence

| `HarnessRun` field   | Vector evidence                                                                                                                  |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `output`             | Final root-session output.                                                                                                       |
| `session`            | Normalized sessions, messages, tool calls and results, counts, availability, ordering, and stable links.                         |
| `traces`             | Run span and complete durable shell start/end pairs. Incomplete pairs do not become spans.                                       |
| `usage`              | Available token and tool-call data. `usage.metadata.opencode.costUsd` appears only with complete, finite authoritative evidence. |
| `errors`             | JSON-safe structured errors exposed by OpenCode or the OpenCode Harness lifecycle.                                               |
| `artifacts.opencode` | Repository diff, sanitized opaque session exports, and bounded process diagnostics.                                              |

> **Isolation is not a sandbox**
>
> The child process keeps the host user's filesystem, network, and process access. Use only trusted Fixtures,
> configurations, tools, and commands.

Execution, observation, evidence, cancellation, and cleanup failures reject with an `OpenCodeHarnessError` carrying a partial `HarnessRun` through vitest-evals' error attachment. Inspect its errors, availability markers, diagnostics, and evidence captured before failure.

See [Harness API](/evaluate/harness-api) for every option and default.