Description

You will build and automate ETL processes and data quality testing frameworks within cloud environments.

Responsibilities

• Implement ETL and data quality automation testing workflows.
• Develop data pipelines using Python, PySpark, and Airflow.
• Manage data processing across GCP services including BigQuery, Dataproc, and Cloud Storage.
• Perform data profiling using Python libraries to ensure data integrity.
• Handle diverse file formats including CSV, ORC, Parquet, JSON, and XML.

Required Skills

• 8+ years of professional experience in data engineering or related roles.
• Strong hands-on experience with SQL and Python.
• Proficiency with PySpark and Airflow.
• Experience implementing cloud services on GCP (BigQuery, Dataproc, GCS, Composer) or equivalent on AWS/Azure.
• Expertise in data profiling using pandas, numpy, scipy, matplotlib, plotly, or seaborn.
• Experience working with JSON and XML formats.
• Ability to work with BigQuery and cloud-native data architectures.

Preferred Skills

• Experience with Great Expectations, ydata-profiling, or scikit-learn.
• Knowledge of Druid, Hive SQL, Flink, or NoSQL databases like MongoDB and Cassandra.
• Familiarity with DevOps, CI/CD, Git, Jenkins, or monitoring tools like Grafana and Prometheus.

Key Skills

Education

Any Graduate