Run an Evaluation Suite

Execute a Vector Configuration in a disposable Fixture and inspect normalized response and effect evidence with vitest-evals.

createOpenCodeHarness() adapts one OpenCode prompt to the HarnessRun contract from vitest-evals@0.16.1. Vitest-evals owns Evaluation Suite cases, judges, execution, and reporting.

Use Node.js 24 on macOS or Linux. Install vitest-evals at 0.16.1 and provide a compatible optional @opencode-ai/cli peer. Commit the Fixture and OpenCode Plugin Entrypoint used by the suite. Treat them and every command they can run as trusted code.

Create the evaluation around your committed Fixture and entrypoint.

import { createOpenCodeHarness } from '@evilkiwi/vector/evals';
import { expect } from 'vitest';
import { describeEval } from 'vitest-evals';

const harness = createOpenCodeHarness({
  fixture: new URL('./fixtures/repository/', import.meta.url),
  plugin: new URL('./fixtures/opencode.ts', import.meta.url),
  agent: 'evaluation-agent',
  model: { providerID: 'provider', id: 'model' },
  environment: {},
});

describeEval('Vector Configuration behavior', { harness }, (it) => {
  it('returns the expected result', async ({ run }) => {
    const result = await run('Complete the Fixture task.');

    expect(result.output).toContain('complete');
  }, 35_000);
});

Run the Evaluation Suite through the repository script that owns your eval files. In Vector’s repository, the command is:

pnpm test:evals

A passing case produces a normal Vitest pass and a HarnessRun available to its assertions and reporters. Each run() copies the Fixture into a disposable temporary repository, starts a fresh supported opencode2 serve process, observes the session through terminal completion, collects evidence, stops the process, and removes temporary state.

Inspect the evidence

`HarnessRun` field	Vector evidence
`output`	Final root-session output.
`session`	Normalized sessions, messages, tool calls and results, counts, availability, ordering, and stable links.
`traces`	Run span and complete durable shell start/end pairs. Incomplete pairs do not become spans.
`usage`	Available token and tool-call data. `usage.metadata.opencode.costUsd` appears only with complete, finite authoritative evidence.
`errors`	JSON-safe structured errors exposed by OpenCode or the OpenCode Harness lifecycle.
`artifacts.opencode`	Repository diff, sanitized opaque session exports, and bounded process diagnostics.

Execution, observation, evidence, cancellation, and cleanup failures reject with an OpenCodeHarnessError carrying a partial HarnessRun through vitest-evals’ error attachment. Inspect its errors, availability markers, diagnostics, and evidence captured before failure.

See Harness API for every option and default.