This a Full Remote job, the offer is available from: South Africa
This is a remote position.
Job Overview:
We are seeking a highly skilled Azure Data Engineer with extensive experience in Databricks, PySpark, and Spark SQL to join our team. The ideal candidate will be responsible for designing, developing, and maintaining large-scale data solutions on Azure, with a focus on real-time and batch data processing using Databricks.
Requirements
Key Responsibilities:
• Design, build, and maintain scalable data pipelines on Azure Databricks using PySpark and Spark SQL.
• Develop and optimize ETL processes for handling large data sets.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and implement solutions.
• Ensure data quality, performance, and scalability by implementing best practices for coding and data architecture.
• Create and manage data models, databases, and data lakes on Azure.
• Monitor and troubleshoot data pipelines... ensuring high availability and reliability.
• Implement security and compliance best practices in the data pipeline, following Azure standards.
• Work with Azure services like Azure Data Factory, Azure Synapse, and Azure Blob Storage to orchestrate and manage data workflows.
Qualifications:
• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 5+ years of hands-on experience with Azure Databricks.
• Proficiency in PySpark and Spark SQL for building scalable data pipelines.
• Solid understanding of Azure data ecosystem, including Azure Data Lake, Azure Data Factory, and Azure Synapse Analytics.
• Experience with ETL processes, data modeling, and data architecture.
• Familiarity with cloud security best practices and data governance.
• Strong problem-solving and troubleshooting skills.
• Excellent communication and collaboration skills.
Nice-to-Have:
• Experience with CI/CD pipelines using tools like Azure DevOps.
• Knowledge of Data Warehousing and Big Data technologies.
• Experience with other programming languages like Python or Scala.
• Familiarity with Machine Learning workflows on Databricks.
This offer from "Advancio" has been enriched by Jobgether.com and got a 72% flex score