Description

Software Requirements

 

• Python: 5+ years of development experience.
• PySpark/Spark: 3+ years of experience.
• Oracle SQL: 3+ years, including query optimization and data analysis.
• Pandas and Jupyter Notebook.
• ETL development and data warehousing.
• Hadoop, Hive, and MapReduce.
• SQL and NoSQL databases.
• Git, CI/CD tools, automated testing, and data validation.
• Experience supporting development, UAT, production deployment, and production issues. 
  
   

Preferred

 

• Cloud-based data platforms.
• Workflow orchestration tools.
• Data quality and monitoring tools.
• Banking or financial services data experience. 
  
   

Overall Responsibilities

 

• Design, develop, and maintain scalable ETL pipelines using Python and PySpark.
• Build and optimize data marts and transformation processes.
• Process structured, semi-structured, and unstructured data.
• Analyze and resolve data issues using Oracle SQL, PySpark, and data analysis techniques.
• Support development, testing, UAT, deployment, and production operations.
• Collaborate with engineering, QA, business, and operations teams.
• Apply coding standards, testing practices, and CI/CD processes.
• Improve data quality, pipeline performance, reliability, and maintainability. 
  
   

Technical Skills (By Category)

Programming Languages

 

• Essential: Python, PySpark, Spark, SQL, and Pandas.
• Preferred: Scala or another data-processing language. 
  
   

Databases and Data Management

 

• Essential: Oracle, SQL and NoSQL databases, data marts, data warehousing, ETL, data validation, and query optimization.
• Preferred: Data modeling, partitioning, indexing, metadata management, and data quality frameworks. 
  
   

Cloud Technologies

 

• Essential: Understanding of distributed data environments and production data platforms.
• Preferred: Experience with cloud storage, compute, managed Spark services, and cloud data warehouses. 
  
   

Frameworks and Libraries

 

• Essential: PySpark, Apache Spark, Pandas, Hadoop, Hive, MapReduce, and Jupyter Notebook.
• Preferred: Workflow orchestration, streaming, and data quality frameworks. 
  
   

Development Tools and Methodologies

 

• Essential: Git, CI/CD, Agile delivery, code reviews, automated testing, debugging, deployment, and production support.
• Preferred: Infrastructure as code, monitoring, alerting, and automated release management. 
  
   

Security Protocols

 

• Essential: Secure handling of data, access controls, secrets management, encryption, and data privacy practices.
• Preferred: Experience with audit controls and regulated data
• environments. 
  
   

Experience Requirements

 

• 5+ years of experience in data engineering or software development.
• Proven experience developing ETL pipelines, data marts, and large-scale data processing solutions.
• Experience with Python, PySpark, Spark, SQL, Oracle, Hadoop, and Hive.
• Experience supporting the complete SDLC, including UAT, deployment, bug fixes, and production support.
• Banking or financial services experience is preferred.
• Equivalent experience in data development, analytics engineering, platform engineering, or production data support may be considered. 
  
   

Day-to-Day Activities

 

• Develop and enhance Python and PySpark data pipelines.
• Build data marts and optimize transformations and SQL queries.
• Investigate data quality issues and pipeline failures.
• Review logs, job results, database records, and performance metrics.
• Create unit, integration, and data validation tests.
• Support UAT, deployments, releases, and production incidents.
• Participate in planning, requirements discussions, code reviews, defect triage, and retrospectives.
• Document data flows, technical decisions, support procedures, and known issues.
• Recommend improvements to pipeline design, performance, testing, and reliability. 
  
   

Qualifications

 

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience.
• Certifications in data engineering, cloud, database, or big-data technologies are preferred.
• Complete Synechron training on security, data protection, SDLC, and operational procedures.
• Maintain current knowledge of Python, Spark, data architecture, cloud platforms, data quality, and engineering practices

Key Skills

Education

Bachelor's degree