Sunil Kumar Reddy
Data Engineer
Summary
Data Engineer specializing in real-time streaming pipelines, cloud-native ETL, and scalable data architectures across AWS, GCP, and Azure. Experienced building fault-tolerant systems processing 100M+ events/day with sub-second latency and exactly-once guarantees. Active Apache Airflow open-source contributor.
Core Skills
Selected Projects
Real-Time Fraud & Anomaly Detection
Engineered a real-time fraud detection streaming platform processing 100M+ transactions/day with sub-2-second latency, improving fraud detection speed and reducing fraudulent transaction risk using Kafka, Flink, PySpark, and AWS.
Python · Kafka · Flink · PySpark · AWS S3 · Redshift · Cassandra · Airflow
Data Quality & Streaming Governance
Developed a real-time data quality and governance platform processing 50M+ streaming records/day, reducing invalid data incidents by 70% using Kafka, Flink, PySpark, and AWS.
Python · PySpark · Kafka · Flink · Great Expectations · Deequ · Airflow · AWS S3
Global Event Processing Platform
Built a globally distributed real-time stateful streaming platform processing 100M+ events/day with sub-2-second latency using Kafka, Flink, PySpark, and AWS.
Python · Flink · Spark · Kafka · Airflow · S3 · HDFS · Terraform
Open Source
Apache Airflow — Add deferrable mode to SFTPOperator (PR #68298, in active review). Implemented an async trigger and native asyncio I/O for non-blocking SFTP transfers, reviewed alongside Airflow committers and PMC members.
Education
Bachelor of Computer Applications (BCA)
Mar 2019 – Jul 2021Maharaja Agrasen Himalayan Garhwal University
Master of Computer Applications (MCA)
Jun 2021 – Apr 2023Maharaja Agrasen Himalayan Garhwal University
Certifications
- AWS Certified Data Engineer – Associate — Amazon Web Services (In Progress, Expected 2026)
- Databricks Data Engineering — Databricks (Skills Track, Ongoing)