Description

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Build and optimize ETL and ELT workflows for structured and unstructured data.
• Develop complex SQL queries and optimize database performance.
• Process large datasets using Apache Spark.
• Integrate data from multiple sources into centralized data platforms.
• Develop data transformation and validation processes.
• Implement data quality checks and monitoring solutions.
• Work with cloud-based data storage and processing services.
• Optimize data processing workflows for performance and scalability.
• Collaborate with data analysts, architects, and application development teams.
• Troubleshoot data pipeline failures and resolve production issues.
• Maintain technical documentation and follow data engineering best practices.

Required Skills:

• 7+ years of experience in data engineering.
• Strong hands-on experience with Python and PySpark.
• Expertise in SQL and database development.
• Experience with Apache Spark and distributed data processing.
• Strong knowledge of ETL/ELT development.
• Experience with AWS, Azure, or GCP.
• Knowledge of data warehousing concepts and data modeling.
• Experience with Apache Airflow or similar orchestration tools.
• Familiarity with Git and CI/CD pipelines.
• Strong analytical and problem-solving skills

Key Skills

Education

Bachelor's degree