> ## Documentation Index
> Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# bt eval

> Run JavaScript and Python evaluation files against Braintrust

Run evaluation files against Braintrust. Supports JavaScript and Python.

<Note>
  `bt eval` is currently macOS and Linux only.
</Note>

## File selection

* `bt eval` — discover and run all eval files in the current directory (recursive)
* `bt eval tests/` — discover eval files under a specific directory
* `bt eval "tests/**/*.eval.ts"` — glob pattern
* `bt eval a.eval.ts b.eval.ts` — one or more explicit files

Files inside `node_modules`, `.venv`, `venv`, `site-packages`, `dist-packages`, and `__pycache__` are excluded from automatic discovery. Explicit paths and globs bypass these exclusions.

## Runtime configuration

<Tabs>
  <Tab title="TypeScript" icon="https://img.logo.dev/typescriptlang.org?token=pk_BdcHD9e5SCW3j1rnJkNyMQ">
    Requires Node.js 18.19.0+ or 20.6.0+. Bun 1.0+ and Deno with Node compatibility mode are also supported.

    By default, `bt eval` auto-detects an available runner (`tsx`, `vite-node`, `ts-node`, `ts-node-esm`, then `deno`), checking project-local binaries before `PATH`. Set one explicitly with `--runner` / `BT_EVAL_RUNNER`:

    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    bt eval --runner vite-node tutorial.eval.ts
    bt eval --runner tsx tutorial.eval.ts
    ```

    `bt eval` automatically resolves locally installed binaries from `node_modules/.bin`, so you can write `--runner tsx` instead of `--runner ./node_modules/.bin/tsx` (for example). If you see ESM or top-level await errors, try `--runner vite-node`.
  </Tab>

  <Tab title="Python" icon="https://img.logo.dev/python.org?token=pk_BdcHD9e5SCW3j1rnJkNyMQ">
    Use `--language python` to force language detection. By default, if `VIRTUAL_ENV` is set, `bt` uses that virtualenv's Python; otherwise it searches `PATH` for `python3` or `python`. To use a specific interpreter, set `BT_EVAL_PYTHON_RUNNER` to its name or path (e.g. `python3.11`). The `--num-workers` flag controls concurrency for Python execution.

    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    bt eval my_eval.py
    bt eval --language python --num-workers 4 my_eval.py
    ```
  </Tab>
</Tabs>

## JavaScript auto-instrumentation

For JavaScript and TypeScript evals, `bt eval` applies Braintrust auto-instrumentation before it loads your eval files. This lets supported AI clients and instrumented libraries create trace spans automatically, including clients constructed inside dependencies.

If the installed Braintrust SDK cannot load the auto-instrumentation hook, `bt` prints a warning and continues without it. To disable auto-instrumentation for a run, pass `--no-auto-instrumentation` or set `BT_EVAL_NO_AUTO_INSTRUMENTATION=1`.

## Sampling modes

Run a subset of your evaluation data as a non-final smoke run to catch obvious regressions before committing to the full dataset.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt eval --first 20 qa.eval.ts          # First 20 examples, non-final
bt eval --sample 20 qa.eval.ts         # Random 20 examples, non-final
bt eval --sample 20 --sample-seed 7 qa.eval.ts  # Reproducible random sample
bt eval qa.eval.ts                     # Full dataset, final
```

<Note>
  When `--first` or `--sample` is used, the experiment summary is labeled as non-final in Braintrust. Omitting both flags runs the full dataset and marks the summary as final.
</Note>

## Flags

| Flag                             | Env var                           | Description                                                                                                                                                                                                          |
| -------------------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-r` / `--runner <RUNNER>`       | `BT_EVAL_RUNNER`                  | Runner binary (`tsx`, `bun`, `ts-node`, `python`, etc.)                                                                                                                                                              |
| `-l` / `--language <LANG>`       | `BT_EVAL_LANGUAGE`                | Force language: `java-script` or `python`                                                                                                                                                                            |
| `--filter <PATTERN>`             | `BT_EVAL_FILTER`                  | Run only evaluators matching the pattern                                                                                                                                                                             |
| `--first <N>`                    | `BT_EVAL_FIRST`                   | Run only the first N examples (non-final smoke run)                                                                                                                                                                  |
| `--sample <N>`                   | `BT_EVAL_SAMPLE`                  | Run a deterministic random sample of N examples (non-final smoke run)                                                                                                                                                |
| `--sample-seed <S>`              | `BT_EVAL_SAMPLE_SEED`             | Integer seed for `--sample`                                                                                                                                                                                          |
| `--param <KEY=VALUE>`            |                                   | Pass a named parameter into evaluators that declare a parameters schema (repeatable; also accepts a JSON object string)                                                                                              |
| `--matrix-param <KEY=V1,V2,...>` |                                   | Run one experiment per Cartesian-product combination of parameter values (repeatable). Requires exactly one evaluator (use `--filter` to select it). Incompatible with `--watch`, `--dev`, and `--list`              |
| `--watch` / `-w`                 | `BT_EVAL_WATCH`                   | Re-run when input files change                                                                                                                                                                                       |
| `--no-send-logs`                 | `BT_EVAL_LOCAL`                   | Run without sending results to Braintrust                                                                                                                                                                            |
| `--num-workers <N>`              | `BT_EVAL_NUM_WORKERS`             | Worker threads for Python execution                                                                                                                                                                                  |
| `--verbose` / `-v`               | `BRAINTRUST_VERBOSE`              | Show full errors and stderr from eval files                                                                                                                                                                          |
| `--list`                         | `BT_EVAL_LIST`                    | List evaluators without running them                                                                                                                                                                                 |
| `--jsonl`                        | `BT_EVAL_JSONL`                   | Output one JSON summary per evaluator (for scripts). See also the global `--json` flag ([overview](/docs/reference/cli/overview#global-flags)), which formats all CLI output as JSON rather than per-evaluator summaries. |
| `--terminate-on-failure`         | `BT_EVAL_TERMINATE_ON_FAILURE`    | Stop after the first failing evaluator                                                                                                                                                                               |
| `--no-auto-instrumentation`      | `BT_EVAL_NO_AUTO_INSTRUMENTATION` | Disable automatic instrumentation for JavaScript and TypeScript evals                                                                                                                                                |
| `--dev`                          | `BT_EVAL_DEV`                     | Start a local web server for browser-based eval development                                                                                                                                                          |
| `--dev-host <HOST>`              | `BT_EVAL_DEV_HOST`                | Host interface for the eval dev server (default: `localhost`)                                                                                                                                                        |
| `--dev-port <PORT>`              | `BT_EVAL_DEV_PORT`                | Port for the eval dev server (default: `8300`)                                                                                                                                                                       |
| `--dev-org-name <NAME>`          | `BT_EVAL_DEV_ORG_NAME`            | Restrict eval dev server access to a specific org                                                                                                                                                                    |
| `--dev-allowed-origin <ORIGIN>`  | `BT_EVAL_DEV_ALLOWED_ORIGIN`      | Additional allowed CORS origin(s) for the eval dev server (repeatable; env var accepts comma-separated values)                                                                                                       |

## Summary output

When using `--jsonl` or reading SSE output, each evaluator summary object includes these fields:

| Field         | Type                                | Description                                                                    |
| ------------- | ----------------------------------- | ------------------------------------------------------------------------------ |
| `runMode`     | `"full"` \| `"first"` \| `"sample"` | How the eval was run                                                           |
| `isFinal`     | `boolean`                           | Whether this is a final (full-dataset) run                                     |
| `runLabel`    | `string`                            | Human-readable description of the run mode                                     |
| `sampleCount` | `number`                            | Number of examples sampled (only present when `--first` or `--sample` is used) |
| `sampleSeed`  | `number`                            | Seed used for random sampling (only present when `--sample` is used)           |

## Parameters

`--param` overrides values for evaluators that declare a `parameters` schema via `loadParameters()` (TypeScript) or `load_parameters()` (Python). This is the same parameters system used by [remote evals](/docs/evaluate/remote-evals), where parameters are version-tracked in Braintrust and appear as UI controls in the playground. See [Create evaluation parameters](/docs/evaluate/write-parameters) for how to define and load parameters.

Each evaluator only receives the keys it declares. Extra keys are silently filtered, so a single command can target multiple evaluators with different schemas without errors.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt eval --param model=gpt-4o --param count=5 my.eval.ts
bt eval --param '{"model":"gpt-4o","count":5}' my.eval.ts
```

Parameters are validated against the evaluator's declared schema before execution. Evaluators without a `parameters` schema are unaffected.

## Parameter matrix

`--matrix-param` works with the same parameters system as `--param`. Specify multiple values for one or more parameters and `bt eval` runs one experiment per combination, naming each `<experiment-name> [key=value, ...]`.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# Sweep a single parameter across three values
bt eval --matrix-param model=gpt-4o,gpt-4o-mini,o1-mini my.eval.ts

# Sweep two parameters — runs 2 × 3 = 6 experiments
bt eval --matrix-param model=gpt-4o,gpt-4o-mini --matrix-param temperature=0.0,0.5,1.0 my.eval.ts
```

For values that contain commas, use a JSON array:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt eval --matrix-param model='["gpt-4o","claude-3-5-haiku-20241022"]' my.eval.ts
```

`--matrix-param` requires exactly one evaluator to be selected. If your file exports multiple evaluators, use `--filter` to narrow down to one. It is not supported for eval files that export `btEvalMain`.

## Passing arguments to the eval file

Use `--` to forward extra arguments to the eval file via `process.argv`:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt eval foo.eval.ts -- --description "Prod" --shard 1/4
```

## Running in CI

Set `BRAINTRUST_API_KEY` instead of using OAuth login:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# GitHub Actions example
- name: Run evals
  env:
    BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
  run: bt eval tests/
```

Use `--no-input` and `--json` for non-interactive output:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
BRAINTRUST_API_KEY=... bt eval tests/ --no-input --json
```