Description

Own the design, implementation, and maintenance of scalable data engineering infrastructure and pipelines to support data science, business intelligence, and strategic analytics.

This role is on-site.

Responsibilities

• Design and build data ingestion pipelines to clean, transform, and aggregate data from heterogeneous sources using PySpark, Spark SQL, and Delta Lake.
• Implement statistical data quality procedures, retention models, and anonymization policies to ensure data integrity and compliance.
• Mentor junior data engineers and establish best practices for data architecture, version control, and agile delivery.
• Collaborate with data science and BI teams to develop data models, manage Databricks Notebooks, and optimize stream processing with Apache Kafka.
• Translate business requirements into technical data solutions, providing strategic recommendations based on insights derived from large-scale datasets.

Required Skills

• 7+ years of overall IT experience with 5+ years in data engineering or ETL roles.
• Strong proficiency in Python, PySpark, and Spark SQL for building and optimizing data pipelines.
• Hands-on experience with Big Data tools including Hadoop, Hive, Sqoop, HDFS, and S3.
• Experience with high-velocity stream processing using Apache Kafka and Spark Streaming.
• Proficiency in managing Databricks environments, including Delta Live Tables, Unity Catalog, and Delta Lake.
• Strong SQL skills with the ability to write intermediate to complex queries.
• Experience with Agile Scrum, Kanban, or SAFe methodologies.

Preferred Skills

• Experience with cloud platforms (AWS, GCP, Azure) and serverless technologies like Lambda and Glue.
• Familiarity with BI tools such as Tableau, PowerBI, or Spotfire, and database solutions like Teradata or BigQuery.
• Experience with messaging platforms like IBM MQ Series or TIBCO EMS.

Key Skills

Education

Bachelor's degree