Responsibilities

  • Lead migration to Azure Databricks and redesign data models using Medallion Architecture.

  • Develop ETL/ELT and data quality pipelines with PythonPySpark, and SQL.

  • Automate deployments with Azure DevOpsGitHub, and Jenkins.

  • Optimize Spark jobs for performance and cost efficiency.

  • Mentor engineers and ensure best practices in data architecture and governance.

Requirements

  • Expert in Azure DatabricksADFDelta Lake, and DevOps pipelines.

  • Skilled in PythonPySparkSQL, and CI/CD automation.

  • Strong knowledge of metadata-driven frameworksKafka/EventHub, and Airflow.

  • Experience across regulated industries (finance, insurance, life sciences) preferred.

Aplikuj