Description

Key Responsibilities:

• Ability to analyze data and support data preparation.
• Design and maintain data pipelines for streaming data, batch processing, and real-time analytics.
• Ability to build data pipelines for different ingestion patterns.
• Apply data modeling concepts to pipelines.
• Implement automated data quality checks and pipeline expectations.
• Automate execution and deployment of pipelines.
• Support data preparation, exploration, and visualization to facilitate data-driven decision-making.
• Work closely with other engineering teams to ensure data pipelines are secure, efficient, and aligned with business requirements.
• Troubleshoot production pipelines and resolve failures.
• Document current and future state data flows.

Qualifications:

• Bachelor’s degree in Computer Science, a technical field, or a related business discipline. Equivalent experience, certifications, or training will be considered.
• 5+ years of experience in designing and delivering secure cloud solutions.
• 2+ years of experience with Databricks including:
  • Lakeflow Connect, Lakeflow Jobs, and Spark Declarative Pipelines
  • Databricks Declarative Automation Bundles (DABs), formerly Databricks Asset Bundles
  • Databricks Unity Catalog
  • Delta Lake
  • PySpark
  • SQL
  • Git/CI/CD
  • Testing, monitoring, and performance optimization
• 2+ years of experience with AWS (preferred).
• Experience with Agile methodology.
• Strong problem-solving, analytical, and communication skills

Key Skills

Education

Bachelor's degree