Pratik BhavsarAI engineer at Cisco

What can one engineer build?

I use AI to take products from prototype to production. I’m enjoying getting better at every step.

Explore the stack

All projects ↓
Five layers of a computer suspended in midair

Token Meter

Cost and context for coding agents

Eval Engineer

Diagnose failures from production traces

BRAG

53.23 on ChatRAG-Bench for BRAG-Qwen2-7B

Agent Leaderboard

30+ models across 500 support scenarios

Production AI field guides

Five books on evals, agents, and retrieval

Selected work

Open source · Splunk

Token Meter

A local dashboard that shows what each coding agent session costs, how much context it uses, and how fast it runs.

Show where the number comes from.

Cost estimates carry an est label, and missing data stays marked as unavailable. The dashboard reads traces on your machine.

Token Meter Sessions dashboard with four coding tasks, estimated costs, context use, and output speed.
Sessions dashboard · Sample data

Claude Code · Codex

Eval Engineer

A skill package for debugging AI apps from production traces. It isolates a failure, scopes the fix, and checks whether the next run improved.

Inspect
Find the first bad step in a trace.
Change
Make one fix tied to the failure.
Verify
Rerun the eval and compare results.

GitHub · 2026 ↗

A year of building.

Public commits to date

  1. 17
  2. 50
  3. 214
  4. 412
Days with a public commit
107
Public, non-fork repos
25
Commits by month
Jan
2
Feb
9
Mar
6
Apr
2
May
17
Jun
14
Jul
30
Aug
67
Sep
67
Oct
119
Nov
79

Research

Public benchmarks

Models keep getting smarter and cheaper. The hard part is knowing whether they work, what they cost, and how to fix them when they don’t.

Agent Leaderboard v2

500 multi-turn support scenarios across five industries. The benchmark checks whether the agent picked the right tools and whether it completed the customer's goals.

Banking · Example from the synthetic dataset

A traveler needs a lost card replaced before a trip and wants to confirm the next mortgage payment.

Hallucination Index

How often do models make unsupported claims when given a few pages, a chapter, or roughly a book of context?

RAG · Context length in tokens
Short
<5K
Medium
5K–25K
Long
40K–100K

Models

Open models for RAG

BRAG

With Ravi Theja

We fine-tuned four small models for retrieval-augmented generation. To fit our compute budget, we tested smaller mixes of training data against NVIDIA’s ChatQA reference.

ChatRAG-Bench · Score out of 100
  • ChatQA-1.5-8BNVIDIA reference · 8K context55.17
  • BRAG-Qwen2-7B128K context53.23
  • BRAG-Llama-3.1-8B128K context52.29
  • BRAG-Llama-3-8B8K context51.70
  • BRAG-Qwen2-1.5B32K context46.43

Experience

Where I’ve worked

I take AI from research to shipped product. I also founded Maxpool, a community for AI engineers, and write Pakodas.

  1. 2026 – now
    Cisco

    Tokenomics and evaluation for AI agents. Token Meter ↗

  2. 2023 – 2026
    Galileo

    Open-source benchmarks and five books on AI evaluation. Acquired by Cisco.

  3. 2021 – 2023
    Enterpret

    Founding engineer. Semantic search, NLP pipelines, and MLOps.

  4. 2020
    Jina AI

    Employee #6. Open-source neural search.

  5. 2017 – 2020
    TaskHuman · Morningstar

    Search, recommendations, and quantitative ML.

Built for fun

Frontier Fighters

A 2D arcade fighter I built for the browser, with pixel-art parodies of the people building AI.

  • 21 fighters
  • Local versus
  • Peer-to-peer online play
Frontier Fighters title screen with Potetron and Sam on either side of the arcade menu

Frontier Fighters

Open in new tab ↗

Click the game to give it keyboard focus.