Description

You will build and manage data pipelines and architectures for enterprise-level systems.

Responsibilities

• Build and optimize data pipelines for ingestion, transformation, and loading from various sources.
• Design, develop, and deploy Spark programs in a Databricks environment to process large data volumes.
• Optimize Databricks jobs for performance, scalability, and bottleneck resolution.
• Implement data management, security, and governance best practices within Databricks.
• Perform code reviews to ensure optimal execution patterns and adherence to standards.

Required Skills

• 5+ years of Python coding experience.
• 5+ years of SQL Server-based development of large datasets.
• 5+ years of experience developing and deploying ETL pipelines using Databricks PySpark.
• Expertise in SQL, including stored procedures and reverse engineering existing processes.
• Experience with Delta Lake, Data Warehousing (OLTP, OLAP), and Dimensional data modeling (Dimensions, Facts).
• Proficiency with structured, semi-structured, and unstructured data.
• Experience with cloud data warehouses such as Synapse, BigQuery, Redshift, or Snowflake.
• Background in cloud-based data architectures, messaging, and analytics.
• Experience leading enterprise-wide Cloud Data Platform migrations.
• BA degree in an engineering or computer science discipline.

Preferred Skills

• Master’s degree in a relevant discipline.
• Cloud certification(s).
• Experience with Airflow.

Key Skills

Education

Any Graduate