Responsibilities
-
Lead migration to Azure Databricks and redesign data models using Medallion Architecture.
-
Develop ETL/ELT and data quality pipelines with Python, PySpark, and SQL.
-
Automate deployments with Azure DevOps, GitHub, and Jenkins.
-
Optimize Spark jobs for performance and cost efficiency.
-
Mentor engineers and ensure best practices in data architecture and governance.
Requirements
-
Expert in Azure Databricks, ADF, Delta Lake, and DevOps pipelines.
-
Skilled in Python, PySpark, SQL, and CI/CD automation.
-
Strong knowledge of metadata-driven frameworks, Kafka/EventHub, and Airflow.
-
Experience across regulated industries (finance, insurance, life sciences) preferred.