Available for opportunities

Sunil Kumar Reddy

I design and operate real-time streaming pipelines, cloud-native ETL systems, and scalable data architectures — moving hundreds of millions of events per day with sub-second latency and exactly-once guarantees.

View Projects Download Resume

Bangalore, India

Kafka

Spark

Airflow

AWS

GCP

Azure

01About

Engineering data systems that don't sleep

0K+

Daily Events Processed

Pipeline Failures Reduced

Faster Analytics Delivery

Cloud Platforms Mastered

Apache Airflow PR (in review)

Degrees in Computer Apps

I'm a Data Engineer based in Bangalore, focused on the systems that keep modern data moving: high-throughput streaming pipelines, cloud-native ETL, and fault-tolerant architectures with exactly-once processing. I work fluently across Kafka, Flink, Spark, and Airflow, deployed on AWS, GCP, and Azure.

I'm also an open-source contributor to Apache Airflow, where I've been working through review with project maintainers to ship a deferrable execution mode for the SFTPOperator.

Mar 2019 – Jul 2021

Bachelor of Computer Applications (BCA)

Maharaja Agrasen Himalayan Garhwal University

GPA 7.55/10 · Data Structures & Algorithms, DBMS, Operating Systems, Computer Networks

Jun 2021 – Apr 2023

Master of Computer Applications (MCA)

Maharaja Agrasen Himalayan Garhwal University

GPA 8.0/10 · Distributed Systems, Big Data Analytics, Data Mining & Warehousing

2023 – Present

Data Engineer

Real-time streaming & cloud data platforms

Designing fault-tolerant streaming pipelines, cloud-native ETL, and exactly-once processing systems.

02Tech Stack

The toolkit behind the pipelines

Hover any hexagon to flip it and reveal my proficiency level. Grouped by where each tool lives in the data lifecycle.

Languages

Python

92%Expert

SQL

90%Expert

Java

72%Proficient

Streaming

Kafka

90%Expert

Flink

82%Advanced

Spark

88%Advanced

Airflow

85%Advanced

Cloud

AWS

88%Advanced

GCP

78%Proficient

Azure

74%Proficient

Data Tools

Databricks

80%Advanced

Delta Lake

78%Proficient

BigQuery

80%Advanced

Redshift

78%Proficient

03Projects

Production-grade streaming platforms

Three end-to-end systems built for scale, fault tolerance, and low latency. Click any card for the full architecture and impact.

Streaming

Real-Time Fraud & Anomaly Detection

Stateful stream processing for instant fraud detection

0M+

transactions / day

sub-second latency

events / second

PythonKafkaFlinkPySparkAWS S3+4

Code

Streaming

Data Quality & Streaming Governance

Validate, monitor & enforce quality on live streams

0M+

records / day

fewer bad records

events / second

PythonPySparkKafkaFlinkGreat Expectations+4

Code

Streaming

Global Event Processing Platform

Exactly-once, fault-tolerant stateful streaming at scale

0M+

events / day

sub-second latency

exactly-once

PythonFlinkSparkKafkaAirflow+5

Code

04Open Source

Contributing to Apache Airflow

I contributed a deferrable execution mode to Apache Airflow's SFTPOperator — making long-running file transfers non-blocking by handing them to Airflow's async Triggerer instead of holding a worker slot. The work has been through several rounds of review with Apache committers and PMC members, and is in active review.

#65480 → continued as #68298In active review

Add deferrable mode to SFTPOperator

Started April 19, 2025 · Active as of June 2026

What I built

Implemented an async SFTPOperationTrigger and wired it into SFTPOperator.execute() via self.defer()
Refactored transfer logic into a single transfer() method on both SFTPHook (sync) and SFTPHookAsync (async) — applying DRY after reviewer feedback
Replaced a sync_to_async wrapper with native async I/O (retrieve_file / store_file / sftp.unlink) for real performance gains
Added asyncio.Semaphore + asyncio.gather for bounded concurrent transfers over a single SSH/SFTP connection
Iteratively renamed classes (SFTPOperatorTrigger → SFTPTrigger → SFTPOperationTrigger) to match Apache's cross-provider naming conventions
Recovered orphaned commits after an accidental force-push using git reflog, and reopened the PR cleanly with full transparency to reviewers

Reviewed alongside@dabla@potiuk@srchilukoori

View Pull Request

sftp_operator.py

 class SFTPOperator(BaseOperator):
     def execute(self, context):
-        self.hook.transfer(self.local, self.remote)
+        if self.deferrable:
+            self.defer(
+                trigger=SFTPOperationTrigger(...
+                method_name='execute_complete',
+            )
         else:
+            self.hook.transfer(self.local, self.remote)

What I learned

Working with senior maintainers (dabla, potiuk, srchilukoori) taught me to defend technical decisions with evidence, not just intuition.
Choosing native asyncio over a thread-pool wrapper after being challenged showed me the difference between async-looking and genuinely async code.
Code review culture in a large OSS project is iterative — each round of CI failures (ruff, newsfragments, docs sync, import ordering) sharpened my attention to detail.
Transparency matters: owning a git mistake openly built more trust with reviewers than hiding it would have.

One of the few freshers actively driving a production-level contribution through review in Apache Airflow — alongside committers and PMC members.

05Certifications