Description

Build and optimize data pipelines using Databricks and Spark technologies in Dallas, TX.

Responsibilities

• Implement Medallion Architecture patterns using Delta Lake and Unity Catalog.
• Develop data ingestion pipelines using Airflow, Kafka, ADF, Glue, or BigQuery.
• Optimize Spark jobs using liquid clustering, photon acceleration, and Z-order.
• Manage data lifecycle tasks including vacuuming and data skipping.
• Address data skew using salting, skew hints, broadcast variables, and adaptive query execution.

Required Skills

• 3+ years of experience in data engineering.
• Expert-level Spark SQL and PySpark proficiency.
• Hands-on experience with Delta Lake and Delta Live Tables.
• Experience with Structured Streaming and Delta Sharing.
• Strong Python programming skills, specifically using Pandas and Pytest.
• Practical knowledge of Data Lake implementation and Unity Catalog.
• Experience with cloud environments including Azure, AWS, or GCP.
• Ability to analyze and optimize queries using explain plans.
• Bachelor's degree in Computer Science.

Preferred Skills

• Experience with Databricks Marketplace and Cleanroom.
• Knowledge of Kafka for real-time data streaming.

Key Skills

Education

Bachelor's degree in Computer Science