Description

Lead the design and migration of enterprise data workloads from AbInitio graphs to DBT jobs. Own the end-to-end development of scalable data pipelines.

Responsibilities

• Design, develop, and deploy data pipelines to process large volumes of data.
• Migrate legacy AbInitio processes to DBT and optimize jobs for big data scalability.
• Implement data management, security, and governance best practices within Databricks.
• Collaborate with data scientists and engineers to translate requirements into production pipelines.
• Perform code reviews to ensure optimal execution patterns and adherence to standards.

Required Skills

• 8+ years of total professional experience.
• 5+ years of hands-on experience with Databricks platform and Data Warehouse Architecture.
• Extensive PySpark coding experience.
• 3+ years of Python coding using NumPy and Pandas.
• Proficiency in SQL, including stored procedures and reverse engineering existing processes.
• Experience with Data Warehousing concepts including OLTP, OLAP, Dimensions, Facts, and Data Modeling.
• Knowledge of Azure Cloud services including ADF, Active Directory, App Services, and ADLS.
• Experience implementing CI/CD pipelines.
• Proven ability to lead enterprise-wide Cloud Data Platform migrations.

Preferred Skills

• Experience with Snowflake, SnowSQL, and SnowPipe.
• Cloud certifications.
• Experience with Reporting.

Key Skills

Education

Any Graduate