Description

Job Description:

We are seeking a skilled Data Engineer with strong experience in Python, PySpark, and Google Cloud Platform (GCP) to design, develop, and optimize scalable data pipelines and cloud-based data solutions. The ideal candidate should have hands-on experience with large-scale data processing, ETL development, and cloud-native technologies.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.
• Build and optimize batch and real-time data processing solutions on GCP.
• Develop data ingestion frameworks from various sources such as APIs, databases, files, and streaming platforms.
• Implement data transformation, cleansing, and validation processes to ensure data quality.
• Work with business stakeholders and data analysts to understand data requirements and deliver solutions.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Design and maintain data models and data warehouse solutions.
• Monitor, troubleshoot, and resolve data pipeline issues.
• Implement CI/CD practices and automation for data engineering workflows.
• Ensure compliance with data governance, security, and best practices.

Required Skills

• Strong programming experience in Python.
• Hands-on experience with PySpark and Apache Spark.
• Strong SQL development and query optimization skills.
• Experience with Google Cloud Platform (GCP).
• Hands-on experience with:
  • BigQuery
  • Cloud Storage
  • Dataproc
  • Cloud Composer
  • Pub/Sub
• Experience in building ETL/ELT pipelines.
• Knowledge of data warehousing concepts and dimensional modeling.
• Experience with Git and version control systems.
• Good understanding of Agile/Scrum methodologies.

Preferred Skills

• Experience with streaming technologies and real-time processing.
• Exposure to DevOps and CI/CD pipelines.
• Experience with Airflow orchestration.
• Knowledge of data quality frameworks and monitoring solutions.
• GCP certification is an added advantage

Key Skills

Education

Any Graduate