Description

You will build and optimize data pipelines using Python and Azure services in Louisville, KY.

Responsibilities

• Develop ETL processes using Python for data transformations, error handling, and logging.
• Load and manage data into Databricks-managed Delta tables.
• Connect to diverse data sources, including databases and streaming data.
• Lead data engineering projects and optimize large-scale data infrastructures.
• Design and implement data models within cloud environments.

Required Skills

• 6+ years of experience in data engineering and building data pipelines.
• Strong proficiency in Python, including classes, objects, methods, dictionaries, loops, Wheel files, and packages.
• Extensive experience with Databricks and PySpark.
• Hands-on experience with Apache Spark and Scala.
• Proficiency with Azure SQL Database, Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake.
• Solid understanding of ETL processes and data modeling.
• Experience with Unix shell scripting and scheduling tools.
• Knowledge of Cloud Warehouses, data lakes, and storage formats.

Preferred Skills

• Experience optimizing ETL processes for large-scale data infrastructures.

Key Skills

Education

Any Graduate