Skill map01
Each row points to the section below that shows it.
| Skill | Where | Proof |
|---|---|---|
| Orchestration | BayRidership | Dagster assets with lineage, sensors and schedules on a shared free VM |
| Batch cleaning | BayRidership | PySpark with an explicit schema: every dropped row gets a reason, plus an audit of rows in and out |
| Data modelling | BayRidership | dbt from staging to an SCD2 station snapshot, facts, dimensions and KPI marts |
| Data quality | BayRidership, SeismicSoCal | 84 dbt data tests on every build; a live quality gate that rejects gap-fill zeros, stuck runs and clipping |
| Warehouse cost control | BayRidership | facts partitioned by date and clustered by station, with a bytes-billed cap per query |
| Streaming ingestion | SeismicSoCal | 19 SeedLink streams in 300 s rolling buffers, processed on waveform time |
| ETL with retries | BearLM | batched embedding with retry and exponential backoff into Chroma, BM25 and SQLite |
| Infrastructure as code | BayRidership | Terraform for every GCP resource; CI is the only thing that can deploy |
BayRidership · a batch pipeline into BigQuery02
A Bay Area transit analytics platform I built end to end. Public BART and Bay Wheels data lands in BigQuery through a Dagster pipeline that runs itself on a free Oracle VM.
- 01
Ingested BART origin-destination files, GTFS and Bay Wheels zips into GCS. A Dagster sensor re-downloads a year only when its Last-Modified changes, and a manifest skips files whose ETag and size already match.
- 02
Cleaned with PySpark: an explicit schema, permissive corrupt-record handling, one drop reason per bad row, and Parquet partitioned by year and month. DuckDB merged Bay Wheels' Ford GoBike and Lyft schemas into one.
- 03
Modelled with dbt: staging, an SCD2 station snapshot, dimensions and facts, then KPI marts. The hourly fact is incremental (the last 35 days plus any date not yet loaded), and a singular test checks that every staged trip date reaches it.
- 04
Ran the same SQL on DuckDB locally and BigQuery in production, with facts partitioned by date, clustered by station and capped on bytes billed.
- 05
Hosted Dagster's daemon and UI as systemd units on a 2-core VM shared with SeismicSoCal, with memory caps and a lower OOM priority so the quake detector keeps priority. Every GCP resource is Terraform.
SeismicSoCal · a live streaming pipeline03
A live earthquake detection system on 19 seismometers across Southern California. Its data side is a 24/7 SeedLink stream and a dataset builder over 26 years of catalogue and waveform data.
- 01
Streamed 19 stations over SeedLink into rolling 3-component buffers and processed everything on waveform time, never wall-clock time, so a replay of archived days runs the exact live code.
- 02
Put a quality gate in front of the model that rejects gap-fill zeros, stuck runs, clipping and lone spikes.
- 03
Kept one station list (
network.py), imported by the dataset builder, daemon, API, scorer and replay. Checkpoints carry it too, so the daemon refuses a model trained on a different network. - 04
Appended training data monthly without ever changing older months, and split every dataset chronologically.
BearLM · an ETL pipeline for retrieval04
A fully local Q&A assistant over UC Berkeley CS and Data Science courses. Its ingestion pipeline rebuilds the whole store from mixed source files.
- 01
Extracted PDFs (pdfplumber), Markdown and Jupyter notebooks, some from cloned course repos, and split them by token length with course, section, source and chunk_id metadata.
- 02
Batch-embedded with retry and exponential backoff. Thousands of requests to a local server ran into Windows socket exhaustion (
WSAENOBUFS); batching lets the run finish instead of dying halfway. - 03
Loaded Chroma vectors, a BM25 index and SQLite for analytics in one pass; the BM25 index is pre-warmed on a background thread at startup and cached.