Description

• Data Pipeline Development: Design, build, and maintain      scalable, reliable ETL/ELT pipelines using Databricks, Spark, SQL, and      Python to process large volumes of data.
• Data Architecture & Modeling: Implement and optimize data      models within the Databricks Lakehouse Platform, focusing on creating and      managing Delta Tables for performance and reliability.
• Technical Expertise: Serve as a subject matter      expert on Databricks, Spark, and Delta Lake. Provide technical guidance on      data engineering best practices, including data validation, performance      testing, and production readiness.
• Process and Workflow      Optimization: Support the standardization and automation of data workflows,      including requirements gathering, architecture planning, development,      quality control, and production deployment.
• Data Quality and Governance: Implement robust data      quality checks and help ensure data governance standards are consistently      met throughout the data lifecycle.
• Collaboration: Work closely with data      scientists, analysts, and business stakeholders to understand their data      needs and deliver high-quality, actionable data solutions.
• Documentation and Knowledge      Management: Develop and maintain comprehensive documentation for data      pipelines, table structures, dependencies, and business logic to ensure      clarity and accessibility.
• Continuous Improvement and      Innovation: Stay current with industry trends and proactively identify and      implement enhancements to data processes, tools, and workflows.
• Cost Management and Optimization: Monitor and optimize data      pipelines and cloud resource usage to ensure cost-effectiveness,      implementing strategies to reduce compute and storage costs.
• CI/CD and Automation: Develop and maintain CI/CD      (Continuous Integration/Continuous Deployment) pipelines for data      workflows to improve deployment frequency and reliability.
• Security and Compliance: Partner with security teams      to implement data encryption, access controls, and data masking techniques      to ensure compliance with data privacy regulations

Key Skills

Education

Any Gradute