Description

You will build and optimize data pipelines for efficient data ingestion, transformation, and loading from various sources while ensuring data quality and integrity.

Responsibilities

• Design, develop, and deploy Spark programs in a Databricks environment to process and analyze large data volumes.
• Optimize Databricks jobs for performance and scalability to handle big data workloads.
• Monitor and troubleshoot Databricks jobs, resolving issues or bottlenecks.
• Implement best practices for data management, security, and governance within the Databricks environment.
• Perform code reviews to ensure adherence to requirements and optimal execution patterns.

Required Skills

• 5+ years of Python coding experience.
• 5+ years of SQL Server-based development for large datasets.
• 5+ years developing and deploying ETL pipelines using Databricks PySpark.
• Experience with enterprise cloud data warehouses including Snowflake.
• Experience in Data warehousing (OLTP, OLAP, Dimensions, Facts, and Data modeling).
• Proficiency in SQL and Python programming, including stored procedures.
• Experience working with structured, semi-structured, and unstructured data.
• Experience designing and developing Enterprise Data Warehouse solutions.

Key Skills

Education

Any Graduate