Description

Design, build, and maintain scalable data pipelines for massive financial datasets using Snowflake, Hadoop, Spark, NiFi, and Databricks. Implement data architectures, optimize workflows, and integrate governance, metadata management, and lineage tracking. Write high-quality, maintainable code in Python and SQL following best practices. Develop automated testing frameworks and validation scripts for ETL processes to ensure accuracy, completeness, and timeliness. Perform unit, integration, and end-to-end testing for data pipelines and schema changes using dbt tests and custom Python utilities. Collaborate with Data Engineers, Product, and Data Science teams to embed testing into the development lifecycle. Participate in agile ceremonies with a focus on quality and delivery. Support production incident response with rapid data validation and root cause analysis. Contribute to team knowledge sharing, mentoring junior engineers, and improving technical standards for data reliability and CI/CD integration.

Responsibilities

• Design and maintain scalable data pipelines using Snowflake, Hadoop, Spark, NiFi, and Databricks.
• Implement data architectures with governance, metadata management, and lineage tracking.
• Develop automated testing frameworks and validation scripts for ETL processes and data transformations.
• Collaborate with cross-functional teams to embed testing into the development lifecycle and support incident response.
• Mentor junior engineers and improve technical standards for data reliability and CI/CD integration.

Required Skills

• 15+ years of experience in data engineering and software development.
• Advanced SQL and experience with relational and NoSQL databases.
• Strong expertise in Snowflake, Hadoop, Spark, Databricks, and Kafka.
• Proficiency in Python for data engineering and test automation.
• Experience with orchestration tools, workflow management, and CI/CD pipelines.
• Proven experience in data testing methodologies, ETL validation, and automated testing frameworks.
• Knowledge of data profiling, anomaly detection, and statistical validation techniques.

Key Skills

Education

Any Gradute