Welcome to Braedyn Thompson's portfolio!
See the roles I'm targeting →

Now playing
Berkeley Lab SeismicSoCal BearLM
ROLE I'M TARGETING

AI Engineer

Build grounded LLM features and measure how well they work.

RAGLangChain · LangGraphLLM evaluationGuardrails
SKILL CHECK . . . OK0.82BearLM recall@1, from 0.560.83RAGAS faithfulness50%SQL agent accuracy, from 27%-23%essay grading error

Skill map01

Each row points to the section below that shows it.

SkillWhereProof
Hybrid retrievalBearLMBM25 + dense vectors, Reciprocal Rank Fusion, a cross-encoder reranker
Grounding and refusalsBearLM, Academy of Testersa confidence gate and an exact refusal string; Testy's similarity cutoff
Retrieval and answer evaluationBearLMa gold-label benchmark, recall@1 0.56 → 0.82; RAGAS faithfulness 0.55 → 0.83
Agents with toolsBayRidershipa LangGraph router, SQL generation with retrieved examples, a forecast tool
GuardrailsBayRidershipsqlglot AST checks, a read-only identity, dry runs, byte budgets
A/B testing an LLM featureBayRidershipexecution accuracy 27% → 50%, McNemar p = 0.004
RAG in productionAcademy of Testersscore-band retrieval over Postgres vectors, GPT-4o at temperature 0, constrained JSON
Reliable model callsKigumi Grouptwo failure classes fixed; integration tests on the real request path

BearLM · local RAG that cites or refuses02

A fully local Q&A assistant over UC Berkeley CS and Data Science courses. Every answer cites the exact PDF page, and it refuses when the material doesn't cover the question.

0.82recall@1from 0.56 dense-only
0.83RAGAS faithfulnessfrom 0.55
0.97Answer relevancyfrom 0.80
$0Per queryfully local

  • 01

    Ran the whole stack on one machine (LangChain, Ollama, Llama 3.1 8B, nomic-embed-text, Chroma), so there are no API keys and no cloud bill.

  • 02

    Combined BM25 for exact terms with dense vectors for paraphrase, fused the two lists with Reciprocal Rank Fusion (k = 60), and let a bge cross-encoder reranker pick the best 4 of 15 candidates.

  • 03

    Gated answers on the reranker's score: chunks below 0.02 are dropped, and if nothing clears the bar BearLM returns an exact refusal string instead of guessing. The prompt is context-only at temperature 0, and the UI shows the rewritten query when rewriting is on.

  • 04

    Added clickable page-level citations and Projects: bring your own PDFs, with answers that fall back to the course corpus only for the courses you tick.

  • 05

    Stated the evaluation's limits: the RAGAS runs are small (n = 6) with a local judge, so the 50-question retrieval ablation is the stronger evidence.

BayRidership · a guarded text-to-SQL agent03

A Bay Area transit analytics platform I built end to end. Ask BayRidership lets anyone question its warehouse in plain English: a LangGraph agent writes the SQL, and layers of guardrails decide whether it runs.

27 → 50%Execution accuracy66 gold-SQL questions
10/10Off-topic questions refusedboth arms
1 GBDry-run cap per queryread-only identity

  • 01

    Built the agent as a LangGraph graph: a router that pre-filters data-changing requests, prompt injection and secrets; SQL generation against a schema card with 5 retrieved example queries and one repair retry; a forecast tool; and an answer that cites the SQL it ran.

  • 02

    Layered the guardrails: a sqlglot AST check allows exactly one SELECT on an allowlist of mart tables, the service account can only read, and BigQuery dry-runs every query and refuses anything over 1 GB.

  • 03

    Added budgets: 10 questions per IP per minute, a 10 GB daily byte budget, and a clear message instead of a 500 when the Gemini quota runs out.

  • 04

    Proved the retrieval step in a pre-registered A/B test, scored with an exact McNemar's test on the questions the two arms disagree on. Arm B won by +22.7 points (p = 0.004), so five examples became the default; near-twin examples in the pool mean unfamiliar questions will gain less.

  • 05

    Streamed each step to the page over server-sent events, with Show SQL and Copy SQL. It's been live on Gemini since 2026-10-16; the 50% is a local Llama 3.1 8B, and Gemini hasn't been scored on the full set yet.

Academy of Testers · a RAG grader and a study assistant04

A full-stack AP/SAT study platform I built as ML & AI Software Developer (Jan 2023 – Jun 2026). A RAG essay grader and an AI study assistant, Testy, share one retrieval layer.

-23%Grading error (MAE)vs. a naive LLM grader
0.62QWK vs. official scoresEnglish Lang + Lit
1,536Dimension embeddingscached in memory

  • 01

    Built the grader pipeline: embed the response, retrieve 16 chunks by cosine similarity, stratify them into 9 low, mid and high anchors, grade with GPT-4o at temperature 0, and return JSON with per-row points and cited ref_ids.

  • 02

    Diversified by score band because nearest-neighbour retrieval returns mostly mid-band essays, which pulled grades toward the middle.

  • 03

    Kept vectors in Postgres as JSON arrays with a small pure-Java cosine engine instead of a vector database; every retrieval is logged with rank and similarity for spot checks.

  • 04

    Gave Testy a similarity cutoff and treated the course as a reference, so it stays grounded when the course covers a question and doesn't refuse reasonable ones when it doesn't.

  • 05

    Cached each subject's parsed embeddings in memory; when the AI call fails, the student's allowance is refunded rather than spent.

Kigumi Group · making an AI assistant reliable05

Backend Software Engineer Intern in Hong Kong, Jun – Sep 2025, on the AI assistant behind KiguLab, a digital wellbeing and AI-ethics curriculum for students.

98%Code coverageup from 90%
2Failure classes fixedconnections · query formulation
2Model familiesGPT + image generation

  • 01

    Traced the request path: the assistant builds a query from a learner's message and sends it to a GPT model for text or an image-generation model for pictures.

  • 02

    Separated two failure classes that looked alike from outside: unstable backend connections that dropped or stalled model calls, and malformed queries that made the models error or answer inconsistently.

  • 03

    Wrote integration tests on the real request path to the models and unit tests on how queries are formed, so connection and request-shape problems show up in testing instead of in front of a student.

  • 04

    Pinned each diagnosed failure with a test; the new unit and integration tests took coverage from 90% to 98%.