Job Summary
A fast-growing technology consulting company is looking for a talented Data Engineer with a strong background in data engineering to join its team. You’ll play a key role in designing, building, and maintaining data pipelines using a variety of technologies, with a focus on the Microsoft Azure cloud platform.
Key Responsibilities
- Design, develop, and implement data pipelines using Azure Data Factory (ADF) or other orchestration tools
- Write efficient SQL queries to extract, transform, and load (ETL) data from various sources into Azure Synapse Analytics
- Utilize PySpark and Python for complex data processing tasks on large datasets within Azure Databricks
- Collaborate with data analysts to understand data requirements and ensure data quality
- Implement data governance practices to ensure data security and compliance
- Monitor and maintain data pipelines for optimal performance, and troubleshoot issues as they arise
- Develop and maintain unit tests for data pipeline code
- Work collaboratively with other engineers and data professionals in an Agile development environment
Must-Have Technical Skills
- Databricks
- Microsoft Fabric
- SQL
- PySpark
- Python
Must-Have Competencies
- Data ingestion, data quality, and transformation pipelines using Databricks and ADF
- Consumption of REST APIs
- Optimization of PySpark jobs, SQL procedures, and views
Nice to Have (Design & Admin Skills)
- Warehouse and lakehouse creation
- Data modeling
- Semantic model design
- Refresh management
- Logging and monitoring
- Roles and permissions management
Role Details
- Position: Senior Software Engineer
- Experience: 6–8 years
- Employment Type: Full-Time
- Openings: 1
- Education: Postgraduate
An equal opportunity employer, committed to diversity and an inclusive environment for all employees.