Welcome to Braedyn Thompson's portfolio!
See the roles I'm targeting →

Now playing
Berkeley Lab SeismicSoCal BearLM
INTERNSHIP · RESEARCH · ESNET

Berkeley Lab

Data Science & Machine Learning Research Intern at Lawrence Berkeley National Laboratory (ESnet), forecasting traffic at the regional data caches that serve CMS physicists, and finding out how predictable that traffic really is.

Sep 2026 – presentBerkeley, CAPatchTST · PyTorch · NeuralForecast
1 · lookback windowforecast horizons2 · split into patches → one token eachp1p2p3p4p5p6p7p83 · Transformer encoder (channel-independent)4 · multi-horizon headsshort horizonmid horizonlong horizonschematic · not real data

Overview01

If the people who run a data cache know traffic will spike, they can provision for it. If they miss the spike, everything downstream slows down.

My work has two halves. One is building a forecaster that beats a hard-to-beat naive baseline. The other is finding out, with evidence, how predictable the peaks are at all, so the team designs features and models around a real limit. I present to the group every week, and each week's question comes out of the last one's result.

  • 01

    Implemented PatchTST (PyTorch / NeuralForecast) for multivariate forecasting of cache traffic at 1-, 7- and 30-day horizons, choosing the lookback window with rolling cross-validation and beating a persistence baseline on every variable and horizon, by up to 24% RMSE.

  • 02

    Showed that the missed traffic peaks are a limit on what past traffic can predict: changing the loss or forecasting high on purpose missed the same spikes, which redirected the work toward new input signals.

  • 03

    Built reproducible Pandas EDA pipelines over several years of XCache logs from three regional caches, and showed that the best amount of history depends on whether traffic is calm or erratic.

The research at a glance

QuestionHow I tested itWhat I concluded
Can a deep forecaster beat the naive baseline at all?PatchTST vs. persistence and linear regression on a chronological hold-outYes on the headline metric, but the average error hid that it smoothed over every spike
How much history should the model see?Lookback sweep per horizon, scored with rolling-origin cross-validationLong context helps a day ahead; at a month ahead short windows win
Is that result real or luck?Hyperparameter grid on the two finalists, then retraining on several random seedsStable across seeds; the long window specifically controls large errors
Are the missed peaks a training problem?Five losses from soft to strict, then deliberately high percentile forecastsNo: every variant misses the same spikes, so the signal isn't in past traffic
Does the right setup depend on the kind of traffic?Separate models for a calm era and an erratic era of the same cacheYes: calm traffic rewards longer history, erratic traffic rewards the last few days

What PatchTST is & how it works02

PatchTST (Nie et al., ICLR 2023, "A Time Series is Worth 64 Words") is a Transformer built for long-horizon forecasting. Given the last L days of a series (the lookback), it predicts the next H days (the horizon) in one shot, instead of predicting one day and feeding that guess back in. One set of weights serves every variable.

Its main idea is borrowed from vision transformers, which cut an image into small squares and treat each square as a "word". PatchTST cuts a time series into short patches of consecutive days and treats each patch as a token.

1 · lookback windowforecast horizons2 · split into patches → one token eachp1p2p3p4p5p6p7p83 · Transformer encoder (channel-independent)4 · multi-horizon headsshort horizonmid horizonlong horizonschematic · not real data
SchematicHow PatchTST sees a lookback window: patches become tokens, a channel-independent Transformer encodes them, and a head emits the forecast. Schematic only, not real data.

How a forecast is made, step by step

01Normalizeeach window is shifted and scaled to mean 0, std 1 (RevIN); undone at the end
02Patchslide a window of P days with stride S, so ~L/S tokens instead of L
03Embedeach patch becomes a vector, plus a learned position embedding
04AttendTransformer encoder: every patch can look at every other patch
05Forecastflatten all patch outputs; one linear layer emits all H future days
  • Normalize per window (RevIN). Traffic levels drift over months. Removing each window's own mean and scale lets the model learn the shape of what comes next, then the level is added back to the output.
  • Patch. A 128-day window with 16-day patches and an 8-day stride becomes about 15 tokens instead of 128. Each token now carries a local shape (a ramp, a weekly dip, a burst) rather than a single number with no context.
  • Attend. Self-attention learns which stretches of the past matter for the forecast: last week, the same point last month, the start of a burst. Its cost grows with the square of the number of tokens, so patching is what makes a long lookback affordable.
  • Channel independence. Each of the six traffic variables goes through the same network separately. That multiplies the training examples and reduces overfitting on a short history, at a cost: the model doesn't know that hits plus misses must equal accesses.
  • Direct multi-step output. The linear head predicts every future day at once, so errors don't compound the way they do when a model feeds its own predictions back in.
A Transformer on daily valuesPatchTST
One token isone day (a single number)a patch of consecutive days
Tokens for a long lookbackone per dayabout one per stride
What attention comparesindividual days, noisylocal shapes, more meaningful
Variablesoften mixed into one tokenencoded separately, shared weights

How I set it up

  • Framework: NeuralForecast's PatchTST on PyTorch Lightning, with RevIN on and early stopping on validation loss.
  • Patches follow the lookback: 16-day patches with an 8-day stride once the window is long enough; for very short windows the patch shrinks to half the window, since a patch can't be longer than its input.
  • Scaling: each variable standardised, because counts and data volumes live on very different scales.
  • Chosen deliberately: lookback, horizon, model size, dropout and loss were each tested as experiments (below).

The problem & the data03

Experiments like the Large Hadron Collider store their data at a few large sites, while the physicists analysing it are spread across the country, and ESnet carries that traffic. Much of a popular dataset is read again and again, so regional caches (XCache) keep copies close to the users: fewer repeated transfers, lower latency, less wide-area traffic. Forecasting how hard each cache will be used is what lets operators size them.

The data is a daily record from three CMS caches (Southern California, Chicago and Boston) covering several years. Each day has six variables: how many file accesses there were and how much data they moved, split into cache hits (served locally) and misses (fetched over the network). Operators care most about the volume served from the cache.

Exploring before modelling

I built reproducible Pandas pipelines that load, clean, resample and plot every site the same way and rerun end to end on new data. What the exploration found:

Structure in the dataTotal accesses always equal hits plus misses, for counts and volumes, a constraint a model should respect.
Sites differThe largest cache has the most traffic and the best hit rate; the smallest is more volatile. One model shouldn't be assumed to fit every site, so sites are modelled separately.
Right-skewed peaksMost days are ordinary and a few are extreme. The rare tail is what operations care about most.
Change-pointsThe level and variance of traffic shift over time, so old history can mislead. This later became its own experiment.

The first model and its baselines04

Step 1 · Baselines and the hold-out split

For traffic, "tomorrow looks like today" (persistence) is a strong baseline because most of the signal is momentum and weekly rhythm. I compared PatchTST with persistence and a linear model on a chronological hold-out (never a random split, which would leak the future), with early stopping on validation loss to avoid overfitting.

Step 2 · Plot the forecasts against the actual traffic

The first model beat both baselines on most variables, including the one operators care about. Plotting the forecasts against reality showed a problem: the model tracked the typical level but smoothed straight through the spikes, and on the low-volume miss variables it collapsed toward zero. Those variables only looked fine because their true values are mostly near zero too.

Conclusion: RMSE rewards being right on average, so a model that plays it safe near the typical value can score well while missing every spike. From then on I judged every experiment on both the average error and the peaks.

Daily hit volume, actual vs PatchTST
FigureDaily cache-hit volume over the test period: actual vs. the first PatchTST model. The forecast tracks the everyday level and rhythm but stays far below the rare spikes.

Choosing how much history to use05

I set each hyperparameter by testing it, and ran the one that mattered most, the amount of history, as an experiment of its own.

LookbackHow many past days the model sees. Swept from a few days to most of a year, separately for each horizon.
Horizon1, 7 and 30 days, each evaluated on its own so a good short horizon can't hide a weak long one.
ScalingEach variable standardised, because counts and data volumes live on very different scales.
LossMAE aims at the median and under-shoots bursts; MSE chases peaks but inflates typical-day error. Tested in between.

Step 3 · Choose the lookback with rolling cross-validation

A single test period can flatter one setting by chance, so every lookback was scored across many rolling windows: train on the past, forecast the next block, slide forward, repeat. The persistence baseline was scored on the identical windows, so every comparison is like for like.

Rolling-origin (walk-forward) cross-validationtrainvalidateunused futurefold 1fold 2fold 3fold 4fold 5time → never train on the future
SchematicRolling-origin cross-validation: each fold trains only on the past and validates on the next block, which mirrors how the model would actually be used.

Walk-forward only: no fold ever peeks at the future.

What I saw: one day ahead, error was lowest with roughly three months of history, long enough to span a monthly cycle. A week to a month of history did worst (too short to see a cycle, long enough to add noise), and among the short windows, 3 days stood out. A week ahead, 3 days was best.

Error above the best lookback, by lookback and horizon
FigureError for each lookback as a percentage above the best lookback at that horizon (0% = best). Computed from the rolling-CV sweeps for 1 and 7 days and the 30-day sweep; relative values only.

At each variable's best setting, PatchTST beat persistence on every variable at both horizons, with the largest gains on data volumes one day ahead and the smallest on miss counts a week ahead.

RMSE improvement over persistence by variable and horizon
FigureRMSE improvement over persistence at each variable's best lookback, on the same rolling windows.

Step 4 · Check the result isn't luck

  • Head-to-head grid. I tuned model size and dropout for the two finalists (3 vs. 100 days of history). On average error their configurations overlapped; on RMSE, which weights big misses, every 100-day configuration beat every 3-day one. So long context specifically controls large errors.
  • Seeds. Retraining the chosen model on several random seeds changed its error only slightly, so the choice is a stable finding rather than a lucky initialisation.
Grid search, error relative to the best configuration
FigureEach configuration's error as a percentage above the best one, averaged over the six variables. Dark = 100-day lookback.
Seed-to-seed spread by variable
FigureSeed-to-seed spread (std / mean) of the chosen model over 5 seeds. Only the low-volume miss_size moves more than a few percent.

Step 5 · Push the horizon out to a month

If long history helps a day ahead because it captures seasonality, it should help even more a month ahead. It didn't: at 30 days the short windows won and the longest ones never did (the dashed line in the lookback chart above).

So a longer lookback is always better?

Not a month ahead. Short windows win there, and the longest ones never do.

But long history wins a day ahead?

Right, and the seed check says that's real. The seasonal advantage just doesn't carry out to 30 days.

Testing why peaks are missed06

Every experiment so far tracked ordinary days well and under-predicted the rare peaks. Rather than keep tuning, I turned the two obvious explanations into hypotheses and tested each directly.

Hypothesis 1 · The loss function is to blame

If training rewards average correctness, a stricter loss should chase peaks. I trained the same model under five losses from softest to strictest (MAE, three Huber settings, MSE). All five produced nearly identical forecasts and missed the same spikes by the same amount. Measured against the MAE-trained model, every stricter loss was equal or slightly worse, on peak days too. Rejected.

Change in error vs. the MAE loss
FigureEach loss compared with the MAE-trained model (0 = same). If stricter losses chased peaks, the dark bars would be negative.
Prediction vs actual for five loss functions
FigurePrediction vs. actual for five training losses (MAE, Huber δ = 0.5 / 1.0 / 2.0, MSE). The shaded areas are the peak gap: under every loss the largest spikes are still under-predicted.

All five losses under-predict the biggest spikes.

Hypothesis 2 · Just forecast high on purpose

For provisioning, over-shooting is cheaper than falling short, so I trained percentile forecasts that aim above the median. Raising the target added spare capacity on ordinary days but barely changed how often a busy day was missed: the whole forecast shifted up without learning to see spikes coming. Rejected.

Conclusion · A limit on what history can predict

If neither a stricter objective nor a deliberately high forecast catches the spikes, the information isn't in the past traffic. The peaks are infrequent and rarely foreshadowed, so a model trained on history alone can't anticipate them. That changed the team's question from "which model predicts peaks?" to "what signal would make peaks predictable?", and it now drives feature design: calendar features, rolling variance and recent miss activity.

Calm vs. erratic traffic07

The most recent test period was far more erratic than earlier years, which raised a new question: is there one best setup at all, or does it depend on the kind of traffic?

Step 6 · Split by regime and compare

I divided the longest-running cache's history into a calmer earlier era and an erratic recent era, trained a separate model on each with the same chronological split, and repeated the lookback-by-horizon comparison inside both.

Calm trafficThe best amount of history grows with the horizon: short windows a day or a week ahead, and long history starts to pay off a month ahead.
Erratic trafficA window of the last few days wins almost everywhere. Rapid shifts make long history stale, so it turns into misleading noise.
Error by eraError was far higher in the erratic era, so a single model judged on mixed data hides how differently it behaves in each.
Heatmaps of error relative to the best lookback, calm vs erratic era
FigureFor each era, every lookback's error divided by the best lookback for the same variable and horizon (1.0 = best, outlined). Ratios only: absolute errors differ hugely between the eras.

Conclusion: rather than one model for all traffic, detect the regime and use a model tuned for it. The next step is a measurable seasonality or regime signal that says in advance whether traffic will be calm or erratic, and fixing patch length and stride for each lookback.

Where it went next08

The first model answered one question: can a Transformer beat persistence? The work since then asked a harder one. What is the best honest next-day forecast, and does anything beyond a single PatchTST actually add to it?

A small blend, chosen the hard way

The reported one-day model is a fixed three-part blend: an hourly-input PatchTST that reads two weeks of hourly traffic and predicts the next day's total, the original daily PatchTST, and a shallow decision tree on the daily features. The hourly signal is where most of the extra accuracy came from. Across the rolling-origin folds it averages about 14.5 TB/day of error against persistence's 17.6, roughly 17% lower, and it wins on most of the held-out folds.

What didn't help

Plenty, and each is written up as a null rather than dropped quietly. Zero-shot time-series foundation models (Amazon's Chronos-2 and Chronos-Bolt) made the same mistakes the blend already made and were worse on busy days, so they earned zero weight. Heavier networks (NHITS, TiDE, an LSTM, TimeXer) and gradient-boosted trees added nothing over the blend either.

Rules fixed before the data was read

Every candidate is pre-registered before a single model is fit. Models are chosen on rolling-origin folds with leave-one-fold-out; a separate block of earlier days can only veto a choice, never make one; and the test window is read only after the choice is locked to a hash. The model's one real weakness is stated plainly, that it under-forecasts the rare record-high days, and the whole pipeline is guarded by 566 automated checks.