Description

Key Responsibilities:

• Design, develop, and maintain scalable ETL pipelines to process and transform large data sets.
• Build efficient and reliable data workflows using Python and Apache Spark.
• Write complex SQL queries for data extraction, transformation, and analysis.
• Collaborate with cross-functional teams to understand data requirements and ensure data quality and consistency.
• Optimize pipeline performance and troubleshoot issues in large-scale data systems.
• Implement best practices in data engineering, version control, and code documentation.

Must-Have Qualifications:

• 5+ years of proven experience building and maintaining ETL pipelines in production environments.
• Proficiency in Python and SQL for data transformation and scripting.
• Strong understanding and hands-on experience with Apache Spark.
• Solid data engineering foundation with attention to scalability, reliability, and performance.
• Strong communication skills and ability to work in collaborative, agile environments.

Nice-to-Have Skills:

• Experience with Azure cloud services, especially Azure Data Factory (ADF) and Databricks.
• Familiarity with Power BI for data visualization.
• Exposure to infrastructure as code tools like Pulumi

Key Skills

Education

Bachelor's degree