AI Product Engineering Notes

Notes from 13 sessions on evals, context, and systems.

AI product engineering is the work of turning a model into a useful product. Evals set the foundation for improving an AI product, but there are many knobs to turn, each with different trade-offs.

The lowest hanging fruit is to optimize retrieval and context. Once that’s done, improve your systems and harness. Consider post-training after the other approaches are exhausted.

I hosted an AI Product Engineering series to explore these topics. I’ve summarized all 13 sessions, organized by theme, with links to the recordings and source materials. The full series is 9.5 hours. The notes take about 20 minutes to read.

All 13 sessions are outlined below. Click the table to read the full post on my website.

If you want to stay on Substack, here are links to the notes from all 13 sessions.

Evals

• Evals for Data Agents

• Cut Classification Costs With a Model Cascade

• Automating Error Analysis

• Case Study: Putting Evals Into Production

Context

• An Intro to Multivector Retrieval

• How to Improve Search Agents

• How to Optimize Retrieval Embeddings

• How to Choose an OCR Model

• Steer AI Writing With Footnotes for Agents

Systems

• Debugging Inference Latency

• Open Weight Model Economics

• The Case for Agent Sandboxes

• Turn Eval Results Into a Better Model

Nearly every improvement in these notes starts with good evals. If you want to work through evals with a live group, the AI Evals course has hands-on exercises and office hours.