Can be summarised into profileBelow corresponding threads

Evaluation methods

1Metric-based

Fast, repeatable regression checks

Examples

Exact match / string match
ROUGE / BLEU
Function-call accuracy
Executable tests (e.g. SQL)

Might miss nuance

Combine with human or
model review

2Human

Expert judgment and calibration

Examples

Blind, randomized review
Rank answers or score 1–5

Challenges

Slow · costly · reviewer disagreement

Make it reliable

Refine the scorecard through review
Show examples of score levels
Set a pass/fail threshold
Aggregate reviewer votes

3LLM-as-a-judge

Model graders · scalable judgment

Pairwise

A or B?

Single answer

Score against
rubric

Reference-guided

Compare to
gold answer

Challenges → Recommendations

Position biasSwap answer order
Verbosity biasControl answer length
Unreliable judgmentsValidate against human labels

Make it reliable

Clear rubric · assessment before score
Prefer pairwise or pass/fail when suitable
  1. Human-labeled
    examples
  2. Validate a
    capable judge
  3. Consistent
    human agreement
  4. Scale, then
    optimize cost

Combine methods. Calibrate with humans. Scale what works.

Design your eval process

1Define eval
objective

What does success look like?

2Collect
dataset

Examples that reflect real use

3Define eval
metrics

How will you measure success?

4Run and
compare evals

Iterate and improve

5Continuously
evaluate

Evaluate changes and new cases

Run evals

Establish a baseline

Inspect failures

Find what needs improving

Make a change

Prompt, model, tools or retrieval

Rerun evals

Use the same test dataset

Compare results

Improvements and regressions

1Trigger

Manual

Slack Mention / DM
Delegate a Linear task
Start in Codex

Automatic

Rising Sentry errors
Aggregated user feedback
Completed Experiments
(Code Cleanup)

2Execution

Environment

Dedicated server per engineer
Isolated sandbox per task
Mobile simulator

Repositories

GitHub
GitLab
Open / update PR

Internal Tools

BigQuery
Sentry
Grafana
GrowthBook

3Validation

Depending on the task, different validation methods may be relevant.

Automated tests & CI

Run relevant tests and pass CI checks.

Visual validation

Agent checks in a simulator or browser. Human testing when needed.

Evals

Compare costs, quality and tool-call precision / recall.

Code review

Address CodeRabbit and engineer feedback, then rerun checks.

4Shipping

Release safeguards

Pass CI Pipeline
PR Approval & Merge
Argo
Progressive rolloutCanary · metric checks · rollback
GrowthBook
A/B ExperimentsFeature flags · guardrail metrics

AI + Engineers

How the way we build software changed

ChatGPT systemA high-level map showing ChatGPT connecting chat, files, memory, plugins, scheduled jobs, work, and Sites. ChatConversation and immediate context
Help me plan this project.
Debug this API error with me.
 FilesDocuments, images, and working material
Summarise this PDF.
Review this technical spec.
 WorkTemporary Linux machine, like Codexrunning on a server
Run this analysis and plot it.
Install Whisper and transcribe this.
 SitesBuild and publish web experiences
Make this a live dashboard.
Publish this internal tool.
 ChatGPT PluginsConnect external apps and services
Which Linear tasks are open?
Did I receive any new emails?
 Send me a daily summary ofnew Slack messages. Scheduled JobsTasks that run later or repeatedly
Remind me tomorrow to do X.
Monitor our pipeline every morning.
 MemoryUseful context across conversations
Continue our work on Project X.
What are my biggest gaps?
 Each morning, rank my openLinear tasks using what youknow about my priorities. Persona EngineerProduct team Grade 8Germany Grade 12United States CollegeFrance
100%