Description

Build and optimize data pipelines using Databricks and Spark technologies in Dallas, TX.

You will implement Medallion Architecture patterns using Delta Lake and Unity Catalog, manage data ingestion via Airflow, Kafka, ADF, Glue, or BigQuery, and optimize Spark jobs using liquid clustering, Photon acceleration, and Z-order. You will handle data lifecycle tasks like vacuuming and data skipping, and address data skew using salting, skew hints, broadcast variables, and adaptive query execution.

Responsibilities

• Implement Medallion Architecture patterns using Delta Lake and Unity Catalog.
• Develop data ingestion pipelines using Airflow, Kafka, ADF, Glue, or BigQuery.
• Optimize Spark jobs using liquid clustering, Photon acceleration, and Z-order.
• Manage data lifecycle tasks including vacuuming and data skipping.
• Address data skew using salting, skew hints, broadcast variables, and adaptive query execution.

Required Skills

• 3+ years of experience in data engineering.
• Expert-level Spark SQL and PySpark proficiency.
• Hands-on experience with Delta Lake and Delta Live Tables.
• Experience with Structured Streaming and Delta Sharing.
• Strong Python programming skills, specifically using Pandas and Pytest.
• Practical knowledge of Data Lake implementation and Unity Catalog.
• Experience with cloud environments including Azure, AWS, or GCP.
• Ability to analyze and optimize queries using explain plans.

Preferred Skills

• Experience with Databricks Marketplace and Cleanroom.
• Knowledge of Kafka for real-time data streaming.

Bachelor's degree in Computer Science.

Key Skills

Education

Bachelor's degree in Computer Science