Description

You will build and develop large-scale cloud data processing systems within the Google Cloud Platform.

Responsibilities

• Design and implement scalable data solutions and re-architect pipelines to build BigQuery datasets for IAM dashboards.
• Ensure data access permission changes reflect in Tableau dashboards within a 24-hour window.
• Build bidirectional integrations between GCP and Collibra and run customized DLP scans.
• Develop metadata extraction pipelines and automate data acquisition processes.
• Collaborate with technical and business users to manage datasets across multiple projects.

Required Skills

• 5+ years of experience in an engineering role using Python, Java, Spark, and SQL.
• 5+ years of experience working as a Data Engineer in GCP.
• Proficiency with Google’s Identity and Access Management (IAM) API.
• Demonstrated proficiency with Airflow and Composer.
• Hands-on experience with GCP tools including BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage, and Bigtable.
• Experience with big data technologies such as HDFS, Spark, Impala, Hive, Kafka, and Flume.
• Strong Linux/Unix background with Shell and Bash scripting skills.
• Experience with CI/CD, Jenkins, Artifactory, and Terraform.
• Proficiency with version control using GitHub and managing tasks in Jira and Confluence.

Preferred Skills

• Coding skills in Scala.
• Experience implementing Dataplex or custom format-preserving encryption.

Key Skills

Education

Any Graduate