Description

Build and maintain scalable data pipelines in a distributed environment. Own the full lifecycle of data processing workflows from design to optimization.

Responsibilities

• Design and develop data pipelines using PySpark.
• Optimize and tune Spark jobs for performance and scalability.
• Write clean, maintainable, and efficient code.
• Perform data validation and quality checks on datasets.
• Troubleshoot and debug issues in data processing workflows.

Required Skills

• 10+ years of professional software engineering experience.
• Strong proficiency in Python.
• Extensive experience with Apache Spark and PySpark.
• Expertise in SQL and relational databases including PostgreSQL and MySQL.
• Understanding of data warehousing concepts and architectures.
• Familiarity with data processing frameworks and ETL tools.
• Bachelor's Degree in Computer Science, Information Technology, or a related field.

Preferred Skills

• Experience with cloud platforms such as AWS, Azure, or GCP.
• Knowledge of data visualization tools including Tableau and Power BI.
• Familiarity with machine learning libraries like scikit-learn or TensorFlow.

Key Skills

Education

Bachelor's Degree