A Docker-deployable HTTP server that implements OpenObserve's Remote Scorer protocol. Runs the eight built-in LLM-Judge dimensions (hallucination, answer_relevance, coherence, helpfulness, instruction_following, toxicity, bias, output_format) entirely inside your environment.

docker run -d \
  -p 8000:8000 \
  -e O2_JUDGE_AUTH_TOKEN=$(openssl rand -hex 24) \
  -e O2_JUDGE_PROVIDER=openai \
  -e O2_JUDGE_API_KEY=sk-... \
  -e O2_JUDGE_MODEL=gpt-5-mini \
  openobserve/judge-server:latest

LiteLLM under the hood, so anything LiteLLM supports works. Common ones below. Set O2_JUDGE_PROVIDER to the provider name and O2_JUDGE_MODEL to the model — that's it.

Concrete examples for each in .env.example.

Anthropic Claude:

docker run -d -p 8000:8000 \
  -e O2_JUDGE_AUTH_TOKEN=$(openssl rand -hex 24) \
  -e O2_JUDGE_PROVIDER=anthropic \
  -e O2_JUDGE_API_KEY=sk-ant-... \
  -e O2_JUDGE_MODEL=claude-3-5-sonnet-20241022 \
  openobserve/judge-server:latest

Local Ollama:

docker run -d -p 8000:8000 \
  -e O2_JUDGE_AUTH_TOKEN=$(openssl rand -hex 24) \
  -e O2_JUDGE_PROVIDER=ollama \
  -e O2_JUDGE_API_KEY=unused \
  -e O2_JUDGE_MODEL=llama3 \
  -e O2_JUDGE_URL=http://host.docker.internal:11434 \
  openobserve/judge-server:latest

AWS Bedrock:

docker run -d -p 8000:8000 \
  -e O2_JUDGE_AUTH_TOKEN=$(openssl rand -hex 24) \
  -e O2_JUDGE_PROVIDER=bedrock \
  -e O2_JUDGE_API_KEY=unused \
  -e O2_JUDGE_MODEL=anthropic.claude-3-sonnet-20240229-v1:0 \
  -e AWS_ACCESS_KEY_ID=... \
  -e AWS_SECRET_ACCESS_KEY=... \
  -e AWS_REGION_NAME=us-east-1 \
  openobserve/judge-server:latest

Three endpoints. Bodies are JSON; auth is Authorization: Bearer <token> matched against O2_JUDGE_AUTH_TOKEN.

Score a single span. Bearer required.

Request:

POST /evaluate
Authorization: Bearer <token>
Content-Type: application/json

{
  "scorer_name": "hallucination",
  "input": "What is the capital of France?",
  "output": "The capital of France is Paris.",
  "expected": "Paris",
  "metadata": {"trace_id": "abc-123"}
}

Response (200 OK):

{
  "value": 0.15,
  "metadata": {
    "reason": "The output's claim that Paris is the capital is supported by the input.",
    "model": "gpt-5-mini",
    "elapsed_ms": 1245
  }
}

Supported scorers:

Errors:

Confirms the server is up and configured. Does not call the upstream LLM — upstream reachability surfaces on the first real /evaluate call as a 502/504. Bearer required.

Response (200 OK):

{
  "status": "ok",
  "version": "0.1.0",
  "backend": "openai",
  "model": "gpt-5-mini"
}

Server version. No auth.

Response (200 OK):

{
  "version": "0.1.0",
  "api_version": "v1"
}

api_version is the major version of the Remote Scorer protocol — clients can use it for compatibility checks.

See ARCHITECTURE.md for the directory layout, per-file responsibilities, startup flow, and the lifecycle of a /evaluate request.

Requires uv. Install once:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then:

uv sync --extra dev
uv run pytest
uv run python -m judge_server  # serves on $O2_JUDGE_HOST:$O2_JUDGE_PORT

This repo hosts 10–15 general-purpose scoring dimensions. Vertical, domain-specific, or language-specific judges are out of scope — contributors are encouraged to fork.