Skill map01
Each row points to the section below that shows it.
| Skill | Where | Proof |
|---|---|---|
| Hybrid retrieval | BearLM | BM25 + dense vectors, Reciprocal Rank Fusion, a cross-encoder reranker |
| Grounding and refusals | BearLM, Academy of Testers | a confidence gate and an exact refusal string; Testy's similarity cutoff |
| Retrieval and answer evaluation | BearLM | a gold-label benchmark, recall@1 0.56 → 0.82; RAGAS faithfulness 0.55 → 0.83 |
| Agents with tools | BayRidership | a LangGraph router, SQL generation with retrieved examples, a forecast tool |
| Guardrails | BayRidership | sqlglot AST checks, a read-only identity, dry runs, byte budgets |
| A/B testing an LLM feature | BayRidership | execution accuracy 27% → 50%, McNemar p = 0.004 |
| RAG in production | Academy of Testers | score-band retrieval over Postgres vectors, GPT-4o at temperature 0, constrained JSON |
| Reliable model calls | Kigumi Group | two failure classes fixed; integration tests on the real request path |
BearLM · local RAG that cites or refuses02
A fully local Q&A assistant over UC Berkeley CS and Data Science courses. Every answer cites the exact PDF page, and it refuses when the material doesn't cover the question.
- 01
Ran the whole stack on one machine (LangChain, Ollama, Llama 3.1 8B, nomic-embed-text, Chroma), so there are no API keys and no cloud bill.
- 02
Combined BM25 for exact terms with dense vectors for paraphrase, fused the two lists with Reciprocal Rank Fusion (k = 60), and let a bge cross-encoder reranker pick the best 4 of 15 candidates.
- 03
Gated answers on the reranker's score: chunks below 0.02 are dropped, and if nothing clears the bar BearLM returns an exact refusal string instead of guessing. The prompt is context-only at temperature 0, and the UI shows the rewritten query when rewriting is on.
- 04
Added clickable page-level citations and Projects: bring your own PDFs, with answers that fall back to the course corpus only for the courses you tick.
- 05
Stated the evaluation's limits: the RAGAS runs are small (n = 6) with a local judge, so the 50-question retrieval ablation is the stronger evidence.
BayRidership · a guarded text-to-SQL agent03
A Bay Area transit analytics platform I built end to end. Ask BayRidership lets anyone question its warehouse in plain English: a LangGraph agent writes the SQL, and layers of guardrails decide whether it runs.
- 01
Built the agent as a LangGraph graph: a router that pre-filters data-changing requests, prompt injection and secrets; SQL generation against a schema card with 5 retrieved example queries and one repair retry; a forecast tool; and an answer that cites the SQL it ran.
- 02
Layered the guardrails: a sqlglot AST check allows exactly one SELECT on an allowlist of mart tables, the service account can only read, and BigQuery dry-runs every query and refuses anything over 1 GB.
- 03
Added budgets: 10 questions per IP per minute, a 10 GB daily byte budget, and a clear message instead of a 500 when the Gemini quota runs out.
- 04
Proved the retrieval step in a pre-registered A/B test, scored with an exact McNemar's test on the questions the two arms disagree on. Arm B won by +22.7 points (p = 0.004), so five examples became the default; near-twin examples in the pool mean unfamiliar questions will gain less.
- 05
Streamed each step to the page over server-sent events, with Show SQL and Copy SQL. It's been live on Gemini since 2026-10-16; the 50% is a local Llama 3.1 8B, and Gemini hasn't been scored on the full set yet.
Academy of Testers · a RAG grader and a study assistant04
A full-stack AP/SAT study platform I built as ML & AI Software Developer (Jan 2023 – Jun 2026). A RAG essay grader and an AI study assistant, Testy, share one retrieval layer.
- 01
Built the grader pipeline: embed the response, retrieve 16 chunks by cosine similarity, stratify them into 9 low, mid and high anchors, grade with GPT-4o at temperature 0, and return JSON with per-row points and cited ref_ids.
- 02
Diversified by score band because nearest-neighbour retrieval returns mostly mid-band essays, which pulled grades toward the middle.
- 03
Kept vectors in Postgres as JSON arrays with a small pure-Java cosine engine instead of a vector database; every retrieval is logged with rank and similarity for spot checks.
- 04
Gave Testy a similarity cutoff and treated the course as a reference, so it stays grounded when the course covers a question and doesn't refuse reasonable ones when it doesn't.
- 05
Cached each subject's parsed embeddings in memory; when the AI call fails, the student's allowance is refunded rather than spent.
Kigumi Group · making an AI assistant reliable05
Backend Software Engineer Intern in Hong Kong, Jun – Sep 2025, on the AI assistant behind KiguLab, a digital wellbeing and AI-ethics curriculum for students.
- 01
Traced the request path: the assistant builds a query from a learner's message and sends it to a GPT model for text or an image-generation model for pictures.
- 02
Separated two failure classes that looked alike from outside: unstable backend connections that dropped or stalled model calls, and malformed queries that made the models error or answer inconsistently.
- 03
Wrote integration tests on the real request path to the models and unit tests on how queries are formed, so connection and request-shape problems show up in testing instead of in front of a student.
- 04
Pinned each diagnosed failure with a test; the new unit and integration tests took coverage from 90% to 98%.