Description

You will design and build scalable batch and streaming data pipelines to ingest data from upstream Equifax cloud systems and support AI/ML applications.

This role is hybrid.

Responsibilities

• Build complex data pipelines using GCP services including BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Functions.
• Design and implement data engineering frameworks to scale development and deployment across the organization.
• Develop and refine prompts for Large Language Models (LLMs) to assist in data cleansing, transformation logic generation, and automated documentation.
• Explore and implement AI agents to automate repetitive data management tasks, monitor data quality, and orchestrate workflows.
• Set engineering standards and best practices for data infrastructure and AI-assisted development.

Required Skills

• 5+ years of experience in data engineering or data architecture.
• Strong proficiency in Python and SQL.
• Experience with at least two GCP big data tools: BigQuery, Dataflow, Dataproc, Pub/Sub, or Cloud Functions.
• Experience with Airflow or Cloud Composer.
• Experience with Vertex AI.
• At least 1 year of experience in a GCP big data environment.
• Bachelor's degree in Computer Science, Information Systems, or a related field.

Preferred Skills

• Experience with prompt engineering and AI-assisted coding tools.
• Familiarity with LangChain or LlamaIndex.
• DevOps experience with Jenkins, GitHub, and Terraform for GCP infrastructure.

Key Skills

Education

Any Graduate