Welcome to Braedyn Thompson's portfolio!
See the roles I'm targeting →

Now playing
Berkeley Lab SeismicSoCal BearLM
ROLE I'M TARGETING

Data Engineer

Build pipelines that ingest messy sources reliably and reproducibly.

DagsterPySpark · dbtBigQueryStreaming
SKILL CHECK . . . OK67.8MBART trip rows in BigQuery84dbt data tests per build19live seismic streams10.4Kchunks per BearLM rebuild

Skill map01

Each row points to the section below that shows it.

SkillWhereProof
OrchestrationBayRidershipDagster assets with lineage, sensors and schedules on a shared free VM
Batch cleaningBayRidershipPySpark with an explicit schema: every dropped row gets a reason, plus an audit of rows in and out
Data modellingBayRidershipdbt from staging to an SCD2 station snapshot, facts, dimensions and KPI marts
Data qualityBayRidership, SeismicSoCal84 dbt data tests on every build; a live quality gate that rejects gap-fill zeros, stuck runs and clipping
Warehouse cost controlBayRidershipfacts partitioned by date and clustered by station, with a bytes-billed cap per query
Streaming ingestionSeismicSoCal19 SeedLink streams in 300 s rolling buffers, processed on waveform time
ETL with retriesBearLMbatched embedding with retry and exponential backoff into Chroma, BM25 and SQLite
Infrastructure as codeBayRidershipTerraform for every GCP resource; CI is the only thing that can deploy

BayRidership · a batch pipeline into BigQuery02

A Bay Area transit analytics platform I built end to end. Public BART and Bay Wheels data lands in BigQuery through a Dagster pipeline that runs itself on a free Oracle VM.

67.8MBART trip rowshourly origin-destination, 2018–2025
26.0MBay Wheels tripstwo source schemas
17dbt models84 data tests
$0To runinside every free tier

  • 01

    Ingested BART origin-destination files, GTFS and Bay Wheels zips into GCS. A Dagster sensor re-downloads a year only when its Last-Modified changes, and a manifest skips files whose ETag and size already match.

  • 02

    Cleaned with PySpark: an explicit schema, permissive corrupt-record handling, one drop reason per bad row, and Parquet partitioned by year and month. DuckDB merged Bay Wheels' Ford GoBike and Lyft schemas into one.

  • 03

    Modelled with dbt: staging, an SCD2 station snapshot, dimensions and facts, then KPI marts. The hourly fact is incremental (the last 35 days plus any date not yet loaded), and a singular test checks that every staged trip date reaches it.

  • 04

    Ran the same SQL on DuckDB locally and BigQuery in production, with facts partitioned by date, clustered by station and capped on bytes billed.

  • 05

    Hosted Dagster's daemon and UI as systemd units on a 2-core VM shared with SeismicSoCal, with memory caps and a lower OOM priority so the quake detector keeps priority. Every GCP resource is Terraform.

SeismicSoCal · a live streaming pipeline03

A live earthquake detection system on 19 seismometers across Southern California. Its data side is a 24/7 SeedLink stream and a dataset builder over 26 years of catalogue and waveform data.

19Live SeedLink streams300 s rolling buffers
50,743Detection windowsUSGS + SCEDC, 2000–2026
6,243Quakes for sizingM2.0–7.1

  • 01

    Streamed 19 stations over SeedLink into rolling 3-component buffers and processed everything on waveform time, never wall-clock time, so a replay of archived days runs the exact live code.

  • 02

    Put a quality gate in front of the model that rejects gap-fill zeros, stuck runs, clipping and lone spikes.

  • 03

    Kept one station list (network.py), imported by the dataset builder, daemon, API, scorer and replay. Checkpoints carry it too, so the daemon refuses a model trained on a different network.

  • 04

    Appended training data monthly without ever changing older months, and split every dataset chronologically.

BearLM · an ETL pipeline for retrieval04

A fully local Q&A assistant over UC Berkeley CS and Data Science courses. Its ingestion pipeline rebuilds the whole store from mixed source files.

10.4KChunks per rebuild10 courses
~40 sBM25 index buildpre-warmed at startup
$0Per queryfully local

  • 01

    Extracted PDFs (pdfplumber), Markdown and Jupyter notebooks, some from cloned course repos, and split them by token length with course, section, source and chunk_id metadata.

  • 02

    Batch-embedded with retry and exponential backoff. Thousands of requests to a local server ran into Windows socket exhaustion (WSAENOBUFS); batching lets the run finish instead of dying halfway.

  • 03

    Loaded Chroma vectors, a BM25 index and SQLite for analytics in one pass; the BM25 index is pre-warmed on a background thread at startup and cached.