Description

About the Role:

We're hiring an exceptional Senior Data Engineer with deep data engineering expertise and strong, hands-on AI skills. You'll build scalable data infrastructure across ingestion, transformation, orchestration, and delivery — collaborating with backend, BI, and ML teams — while actively using AI tooling (Claude, Cursor, and beyond) as a core part of how you engineer and ship. Strong ownership mindset, data quality focus, on-call reliability, and a genuine drive to adopt modern AI tooling are a must.

Responsibilities:

· Design, build, and maintain scalable data ingestion and processing pipelines using SQL , Spark, Hadoop, Airflow, SnapLogic, or similar tools

· Optimize existing ETL/ELT workflows and resolve bottlenecks in production jobs

· Integrate multiple structured and unstructured data sources from internal and external systems

· Design and implement event-driven data pipelines using event bus architectures (Kafka, AWS EventBridge, SNS/SQS, or similar)

· Build and maintain event producers, consumers, and stream processors for real-time and near-real-time data flows

· Manage event schema design, schema registry, and versioning to ensure backward compatibility across consumers

· Monitor event bus health, lag, throughput, and reliability; respond to failures before SLA breach

· Implement and manage CI/CD pipelines for data workflows using Git, Jenkins, or GitHub Actions

· Ensure robust job scheduling, monitoring, logging, and end-to-end testing of data pipelines

· Debug and resolve production issues before SLA breach; participate in on-call rotation

· Collaborate with backend and BI teams for API and data integration

· Design and develop schema definitions; support data warehouse/mart to integrate disparate data sources and make them available for analysis

· Support large data volumes and flexible provisioning of new data sources

· Contribute to data architecture design across the data landscape

· Write clean, reusable, well-tested Python and Shell code across the full software development lifecycle

· Gather functional requirements, develop technical specifications, and own project and test planning

· Work in Agile/Scrum environments; help align team priorities across the organization

· Leverage AI-assisted dev tools (Claude, Cursor .. etc) to accelerate delivery and code quality

Required Skills:

· Strong Python, Advanced SQL, PySpark or Scala, Unix scripting

· Data orchestration: Airflow, SnapLogic, DBT, or equivalent

· Event bus & streaming: Kafka, AWS EventBridge, SNS/SQS, Kinesis, or similar

· Event-driven architecture: producers, consumers, schema registry, dead-letter queues

· AWS cloud: EC2, EMR, S3, Athena, Glue; Databricks

· Data lake & warehouse: Snowflake, Delta Lake, Redshift, BigQuery

· CI/CD tooling: Git, Jenkins / GitHub Actions, Docker

· Experience with APIs for data exchange and integration

· Data quality mindset, ownership, and SLA awareness

· NoSQL, Neo4j, GraphQL is plus

· Agile/Scrum development experience

Preferred / Nice to Have:

· Experience with AI/ML models and feature pipelines

· Exposure to BI platforms (Tableau, Power BI) or data serving for dashboards

· Spark Streaming or Flink for stateful stream processing

Skills:

SQL, AWS, PySpark, Data Engineering, Python PySpark, ETL & Data Analytics, Data Migration, Confluence, Jira

Key Skills

Education

Any Graduate