Overview01
This internship has two research problems. In both, I started by finding out what the data could support.
- 01
Diagnosed the root cause of a stalled inverse-FEA predictor by benchmarking 7 ML models (Random Forest, XGBoost, HGB/ExtraTrees ensembles, PCA pipelines), running paired clean-vs-noisy analysis and a noise-identifiability sweep. This proved the R² ≈ 0.10 ceiling was a data limitation rather than a model limitation, gave the team the measurement precision each target needs, and redirected them away from futile model tuning.
- 02
Raised earthquake detection to 0.9998 ROC-AUC (vs. 0.816 STA/LTA) and magnitude R² to 0.951 (vs. 0.886) by training PyTorch CNN→Transformer and CNN→GNN→Transformer ensembles on 50,743 windows and 6,243 quakes from 19 live stations, with chronological splits and event-clustered bootstrap CIs.
- 03
Built and deployed a real-time SeedLink pipeline that detects, picks, locates (≥ 3 stations, negative evidence) and sizes quakes, then sends two-stage FCM alerts. On a replay of 80 held-out days: 145 push alerts, every one a real quake (chance baseline 0%), median location error 3–4 km, and 83% of isolated M3+ quakes caught.
Diagnosing the inverse-FEA ceiling02
An inverse finite-element problem runs simulation backwards: given a structure's measured response, recover the material properties that produced it. Here, the measured shape of a deformed structure (landmark positions, summarised with PCA) is used to predict the stiffness of several of its components. Two targets were predicted well. One was stuck at R² ≈ 0.10, which was holding up the team's IEEE research paper, and the instinct was to keep tuning models. I set out to find out whether tuning could help at all.
This work is part of an unpublished paper, so its figures and detailed results are held back until publication. The method is below.
Step 1 · Make the evaluation trustworthy
Before comparing models I fixed how they were judged, so a small difference couldn't be an artefact:
I also dropped the chained setup (feeding one target's prediction into the next). The targets are drawn independently, so chaining only passes noise along.
Step 2 · Benchmark seven models
If the bottleneck is the model, different inductive biases should give different answers. I benchmarked seven model configurations under the same cross-validation: a baseline random forest, chained random forests, and my own models, including boosted / ExtraTrees voting ensembles and PCA pipelines. Every one landed on the same ceiling for the hard target, while the easier targets were predicted well by all of them.
Seven very different models hit the same ceiling.
Step 3 · Paired clean vs. noisy data
The decisive test was to train on clean data and on noisy data side by side. If a target is recoverable from clean data but not once realistic noise is added, then no model trained on noisy data will get it back, however it's tuned. That's exactly what happened.
Step 4 · Rule out the obvious bottlenecks
- Is PCA throwing the signal away? I bypassed it and trained on the raw landmark coordinates, alone and alongside the PCA features. Neither rescued the hard target, so the signal isn't hiding in the discarded components.
- Are the noisiest inputs drowning it? I measured each landmark group's noise relative to its signal and ran an ablation on only the cleanest group. Dropping the noisiest inputs didn't rescue the target either.
Step 5 · Sim-to-real
The realistic deployment is a model trained on clean simulations and used on noisy measurements. Trained that way it failed on every target until I mean-centred the noisy features onto the clean ones, which fixed a systematic offset and recovered the easy targets. The hard target stayed unrecoverable.
Step 6 · Noise sweep: how much noise each target tolerates
Finally I rebuilt the feature generator, injected Gaussian landmark noise at increasing levels, re-projected through the clean PCA and cross-validated each target. The result is an identifiability curve: the noise level at which each target stops being predictable. At the real measurement noise the hard target is far past that point, and the curve says how precise the measurements would need to be for it to work.
Could an eighth model have fixed it?
No. On clean data the target is recoverable. With the real measurement noise, no model gets it back.
So what does the team do instead?
Fix the measurements. The noise sweep says how much cleaner they need to be.
The outcome
Every line of evidence pointed the same way: the R² ≈ 0.10 ceiling comes from the data. That redirected the team from model tuning to the inputs: what's measured, and at what signal quality. In the meantime, the useful outputs are the targets that are identifiable, plus a calibrated prediction interval (quantile models) for the hard one instead of a misleading point estimate.
Building SeismicSoCal03
The second half of the internship is SeismicSoCal, deep-learning seismology for Southern California, deployed live. It listens to 19 stations in real time, detects, locates and sizes each quake, and sends two-stage alerts to people who follow nearby sensors. Everything measured offline is exactly what runs live, and a replay of real archived days is the acceptance test.
| Replay of 80 held-out days | Result |
|---|---|
| Push alerts | 145, every one a real quake (chance baseline 0%) |
| Confirmed events that were real | 86% on busy days, 79% on random days (chance 4% / 0%) |
| Median location error | 3–4 km |
| Pushed magnitudes vs. catalogue | MAE 0.12 |
| Isolated M3+ quakes caught | 83% with 3+ sensors online |
It also estimates the shaking at each user's home (MMI), validated on USGS "Did You Feel It?" reports (94% within one intensity level), and it is a production ML system: QuakeOps tracks every training run in MLflow, only promotes a retrained model through a six-rule statistical + replay gate, and checks the live stream for drift every day.
The case study covers the live pipeline, the replay acceptance test and QuakeOps.
How I approach research04
Both halves of this internship (and my other research) follow the same loop. Here is each step, with what it looked like in practice.
The eight steps run as a loop.
- 01
Frame the question
Turn a vague problem into one that evidence can answer.
- CBU: is the R² ≈ 0.10 ceiling a model problem or a data problem?
- SeismicSoCal: not "is the test AUC high?" but "would this live system have alerted correctly on real days?"
- 02
Know the field & pick baselines
Start from what practitioners already use, so a result means something.
- STA/LTA for detection, amplitude + distance for magnitude, and a time-shifted chance baseline for every live precision.
- Persistence for forecasting (Berkeley Lab); a baseline random forest and predict-the-mean for inverse-FEA.
- 03
Design the experiment
Decide how results will be judged before running anything.
- Chronological 70/15/15 splits, hard negatives split by date, thresholds calibrated on validation days and test days scored once.
- Leak-free, out-of-fold cross-validation, and a paired clean-vs-noisy design that isolates noise as the one variable.
- 04
Collect & check the data
Most bad results start as bad data.
- 50,743 SCEDC waveform windows and 6,243 quakes labelled against the USGS catalogue, on one shared station list.
- A quality gate that drops gap-fill zeros, stuck runs, clipping and glitch spikes; three years of XCache logs explored before modelling.
- 05
Run experiments & ablations
Change one thing at a time and see what actually carries the result.
- A nearest-single-station ablation (R² 0.951 → 0.808) that proves the graph fusion matters; a 10-seed study; five alert variants replayed over 20 days.
- A lookback sweep across forecast horizons, and an ablation on only the cleanest inputs for the inverse-FEA model.
- 06
Analyze honestly
Use metrics that can't be gamed, and write down the limits.
- ROC-AUC and MCC on imbalanced data, event-clustered bootstrap CIs, and precision always next to its chance baseline.
- Known limits stated plainly, e.g. off-network quakes are located from one side, and sizes below M2 read high.
- 07
Conclude & communicate
Say exactly what the evidence supports, to the people who need it.
- The inverse-FEA ceiling is a data limit, with the measurement precision each target needs: written up for the team's IEEE paper.
- SeismicSoCal's results published with their baselines and CIs on the live site and its /health page.
- 08
Iterate
Every answer sets up the next question.
- Score shadow mode against USGS before switching live pushes on; run the first gated monthly retrain.
- Test a ~120-day lookback at the 30-day horizon.
Step 8 feeds straight back into step 1.