Data Engineer
- Owned the architecture and system design of a lakehouse data platform and built it from scratch on Kafka, Flink, Spark, Iceberg, and S3, with unified ingestion, transformation, and serving layers for 130 TB/month of web traffic events.
- Reduced end-to-end data latency from 5 minutes to 15 seconds by redesigning the ingestion pipeline as Kafka and Flink streaming jobs writing to Iceberg.
- Migrated Iceberg object storage from MinIO to SeaweedFS after MinIO discontinued support for its community edition.
- Made the platform the single source of truth for four teams (engineering, analytics, support, and marketing), serving analytics, BI, and machine learning workloads.
- Implemented a unified data layer with role-based access control, and established data contracts and pipeline monitoring.