Job Description

Job Title: Databricks Data Engineer Azure Data Lake & ETL

Location: Canada ( Remote )

Duration/Term: Long Term Contract

Job Description

We are seeking an experienced Databricks Data Engineer to design, develop, and maintain scalable data pipelines on Azure Databricks and Azure Data Lake. The ideal candidate will integrate data from various sources, implement robust ETL processes with Azure Data Factory, and optimize data workflows using PySpark. You will collaborate with data scientists and analysts to support data-driven decision-making and ensure high data quality and integrity. Additionally, you will support business use cases such as Bloomberg data acquisition and transformation, and assist with dashboarding and visualization using Power BI.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Lake .
• Integrate data from diverse sources into the Databricks platform to create a unified data ecosystem.
• Implement data integration and ETL processes using Azure Data Factory .
• Develop and optimize data processing workflows and pipelines using PySpark .
• Support business use cases involving Bloomberg data acquisition and transformation.
• Collaborate with data scientists and analysts to facilitate data-driven decision-making.
• Ensure data quality and integrity across various sources and storage solutions.
• Monitor and troubleshoot data pipeline performance and reliability.
• Assist with dashboarding and data visualization using Power BI .

Qualifications

Must-Have:

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field (or equivalent practical experience).
• Proven experience designing and maintaining data pipelines using Azure Databricks and Azure Data Lake .
• Strong proficiency in Python and PySpark for data processing.
• Hands-on experience with Azure Data Factory for ETL processes.
• Familiarity with data visualization tools such as Power BI .
• Excellent analytical, troubleshooting, and problem-solving skills.
• Strong collaboration and communication skills.

Key Skills

Azure Databricks, Azure Data Lake, Python, PySpark, Azure Data Factory, ETL, Data Pipelines, Bloomberg Data Transformation, Power BI, Data Integration, Data Quality, Troubleshooting.

Job Tags

Similar Jobs