Skill map01
Each row points to the section below that shows it.
| Skill | Where | Proof |
|---|---|---|
| Causal inference | BayRidership | pre-registered difference-in-differences against 34 control stations, with placebo intervals |
| A/B testing and power | BayRidership | a power analysis before arm B ran; exact McNemar p = 0.004 |
| Forecast uncertainty | BayRidership | quantile LightGBM with a split-conformal band; 73% coverage reported against an 80% target |
| Validation without leakage | Berkeley Lab, SeismicSoCal | rolling-origin CV; chronological splits; thresholds frozen on validation days |
| Hypothesis testing | Berkeley Lab | two explanations for missed peaks (five losses, percentile forecasts), both rejected |
| Identifiability | CBU Research | a paired clean-vs-noisy design and a noise sweep |
| Confidence intervals | SeismicSoCal | event-clustered bootstrap; a 10-seed study separates seed from sampling variance |
| Probabilistic models | Academy of Testers | BKT + IRT over 8 skills; the essay grader scored by QWK |
BayRidership · causal study, A/B test, forecasts02
A Bay Area transit analytics platform I built end to end. Three of its questions needed statistics: did the eBART extension bring new riders, do retrieved examples make the SQL agent better, and how sure is a two-week forecast?
- 01
Chose the unit to separate new riders from switchers. A rider who switches from Pittsburg/Bay Point to Antioch leaves the line's total unchanged; a new rider raises it. So the study compares the whole line and the three old stations with 34 far-away stations, with station and week fixed effects on log weekly entries: +4.2% for the line, −23.9% for the old stations.
- 02
Led with placebo inference: reassign the extension to all 5,984 sets of 3 control stations and see how often noise alone looks this big (RI p = 0.0007). Intervals from clustered standard errors are reported too, but with only 3 treated stations they under-cover on synthetic data, so the placebo intervals are the headline.
- 03
Pre-registered the design in a hashed file, then validated it on 200 simulations, which caught a too-narrow interval (coverage 0.84, fixed to 0.925 by one dated amendment). The post-hoc trend check gives +1.0% and is labelled post-hoc.
- 04
Ran the agent A/B by the book: a frozen question set, a power analysis after arm A (+14 points detectable at 80% power), a three-part decision rule, then arm B. It won by +22.7 points (95% CI +9.1 to +36.4); near-twin examples in the pool mean unfamiliar questions will gain less, and the write-up says so.
- 05
Back-tested the forecasts walk-forward with a paired bootstrap CI on the improvement (200 to 340 entries a day), and named the band a “calibrated model range” because it held 73%, not 80%.
Berkeley Lab · forecasting and testing why peaks are missed03
Data Science & ML Research Intern at Lawrence Berkeley National Laboratory (ESnet), Sep 2026 – present, forecasting traffic at the regional data caches that serve CMS physicists. I test how predictable that traffic is, peaks included.
- 01
Compared PatchTST with persistence and linear regression on a chronological hold-out, never a random split, which would leak the future.
- 02
Noticed RMSE hid the real failure: the model tracked the everyday level but smoothed through every spike, so I scored each experiment on both average error and peaks.
- 03
Picked the lookback with rolling-origin cross-validation, then checked it wasn't luck: on RMSE every 100-day configuration beat every 3-day one, and re-runs across 5 seeds barely moved.
- 04
Turned the missed peaks into two hypotheses (the loss is to blame; forecast high on purpose) and rejected both. The information isn't in past traffic, so the team's question became what signal would make peaks predictable.
- 05
Split one cache's history into a calm and an erratic era: calm traffic rewards longer history, erratic traffic the last few days. The answer is to detect the regime and use the model tuned for it.
CBU Research · proving a data ceiling04
ML Research Intern at California Baptist University, May 2026 – present.
A model predicting a structure's stiffness from its measured shape (an inverse finite-element problem) was stuck at R² ≈ 0.10 on one target. I tested whether any model could do better.
- 01
Audited the evaluation first (leak-free scaling, out-of-fold scores, a predict-the-mean baseline, the overfitting gap) and dropped the chained setup, which passed noise from one target to the next.
- 02
Benchmarked seven configurations, from a baseline random forest to boosted and ExtraTrees ensembles and PCA pipelines. All hit the same ceiling on the hard target.
- 03
Isolated noise as the one variable with a paired clean-vs-noisy design: the target was recoverable from clean data and gone once realistic noise was added.
- 04
Ruled out the obvious bottlenecks: bypassing PCA and training on only the cleanest inputs didn't rescue it.
- 05
Trained on clean simulations and tested on noisy measurements, then swept the noise level to get an identifiability curve. For the hard target a quantile prediction interval is more useful than a point estimate.
SeismicSoCal · deep vs. classical, with confidence intervals05
A live earthquake detection system on 19 seismometers across Southern California, built during my CBU internship. Each model is measured against the classic seismology method it replaces.
- 01
Split every dataset chronologically (70/15/15), the 2,062 hard negatives of real live-stream noise included, so the test set is the future.
- 02
Bootstrapped confidence intervals by event, so a quake's correlated station windows resample together. The deep and classical intervals don't overlap.
- 03
Ran a nearest-station ablation: with only the closest station, magnitude R² falls to 0.808, worse than the classic formula. The accuracy comes from combining stations.
- 04
Separated seed variance from sampling variance with a 10-seed study. Ensembling added only +0.003, inside the sampling CI, so the live system keeps 5 seeds.
- 05
Fitted the shaking-intensity offset on the older 29 quakes and scored it once on the newer 28 (1,892 report cells). The ground-type term gave no measurable gain, and I report that too.
Academy of Testers · modelling what a student knows06
A full-stack AP/SAT study platform I built as ML & AI Software Developer (Jan 2023 – Jun 2026). Its adaptive SAT engine estimates what each student knows and picks the question that will tell it the most.
- 01
Tracked mastery with Bayesian Knowledge Tracing and damped each step (25% of a gain, 50% of a loss), so one lucky guess can't fake mastery; a guard makes sure a wrong answer never raises a skill.
- 02
Chose the next question with IRT: each item's 3PL Fisher information at the student's ability, blended with the skill gap and recency, then sampled from the top 5 with a softmax.
- 03
Modelled forgetting as exponential decay toward a 0.20 floor (λ = 0.0112/day, so the part above the floor halves in about 62 days).
- 04
Designed the essay-grader A/B so nothing leaks: test essays from 2023/2024, the retrieval corpus from 2025.
- 05
Withheld difficulty until after submission, because knowing it changes how students answer and would contaminate the data used to recalibrate it.