Description

Key Responsibilities

Design, develop, and maintain scalable and reliable data pipelines on GCP.

Build and orchestrate ETL/ELT workflows using Apache Airflow / Cloud Composer.

Develop data processing and transformation solutions using Python and PySpark.

Work with GCP data services to ingest, process, transform, and store large volumes of data.

Develop optimized and fault-tolerant batch and data processing pipelines.

Implement data quality checks, validation, monitoring, and error-handling mechanisms.

Optimize data pipelines for performance, scalability, and cost efficiency.

Troubleshoot pipeline failures and resolve data processing issues.

Collaborate with Data Scientists, Analysts, Architects, and other engineering teams to deliver data solutions.

Follow best practices for coding, testing, deployment, security, and data governance.

Participate in technical discussions, solution design, code reviews, and production support.

Required Skills

7–8 years of experience in Data Engineering.

Strong hands-on experience with GCP and cloud-based data engineering.

Excellent knowledge of Python programming.

Strong experience with PySpark and distributed data processing.

Hands-on experience with Apache Airflow / Cloud Composer for workflow orchestration.

Strong understanding of ETL/ELT concepts and data pipeline development.

Experience working with large datasets and performance optimization.

Good understanding of data warehousing and data processing concepts.

Strong analytical, troubleshooting, and problem-solving skills.

Good communication and collaboration skills.

Good to Have

Experience with BigQuery, Cloud Storage (GCS), Pub/Sub, Dataflow, Dataproc or other GCP data services.

Knowledge of SQL and relational/non-relational databases.

Experience with CI/CD and Git-based development practices.

Understanding of cloud security, IAM, monitoring, and logging.

Experience working in Agile/Scrum environments

Key Skills

Education

Any Gradute