Data Engineer (Azure Expertise) (CB30FT RM 3695)

Position: Data Engineer (Azure Expertise) (CB30FT RM 3695)

Work mode : Hybrid / work from office

Position Overview:
We are seeking a highly skilled Data Engineer with a strong background in Azure technologies and expertise in data pipeline development, transformation, and analytics. The ideal candidate will have hands-on experience with Azure Data Factory (ADF), Azure Functions, Synapse Analytics, SQL Server, Python, and PySpark, along with a solid understanding of CI/CD pipelines and Power BI. Your role will involve designing, building, and maintaining scalable and reliable data solutions that empower the organization to make data driven decisions.

Key Responsibilities:
Design, Build, and Maintain Data Pipelines:

• Develop and optimize end-to-end ETL/ELT pipelines using Azure Data Factory (ADF).
• Integrate, transform, and load data from diverse sources into Azure Synapse Analytics or SQL-based data stores.
• Implement data workflows and automation using Azure Functions.
  Database Development and Optimization:
• Design and manage data models and schemas in SQL Server and Synapse Analytics.
• Write optimized SQL queries and stored procedures for efficient data retrieval and transformation.
• Ensure database integrity, performance tuning, and scalability.
  Big Data Processing:
• Use PySpark to process and analyze large-scale data in a distributed environment.
• Implement and manage data lakes and data lake houses for batch and real-time processing.
  Data Governance and Security:
• Implement data governance policies, including data quality, lineage, and compliance.
• Secure data pipelines using role-based access control and encryption standards.
  CI/CD and Automation:
• Build and manage CI/CD pipelines for data integration and deployment using tools like Azure DevOps, Git, or GitHub Actions.
• Automate testing, version control, and deployment of data engineering solutions.
  Data Visualization and Reporting:
• Collaborate with business stakeholders to design and deliver interactive dashboards using Power BI.
• Enable data exploration and self-service reporting through semantic layers.
  Collaboration and Communication:
• Work closely with cross-functional teams including analysts, and business users to gather requirements and implement data solutions.
• Document workflows, data models, and architecture to ensure knowledge sharing across teams.

Required Skills and Qualifications:
Azure Expertise:

• Hands-on experience with Azure Data Factory (ADF) for building ETL/ELT workflows.
• Proficiency with Azure Synapse Analytics for data warehousing and analytics.
• Strong knowledge of Azure Functions for building event-driven automation.
  Programming:
• Proficiency in Python, with experience in libraries such as Pandas, NumPy, and PySpark.
• Experience with distributed data processing frameworks like Apache Spark.
  Database Management:
• Expertise in SQL Server.
• Knowledge of designing and implementing data lakes and lake houses.
  CI/CD:
• Experience building and deploying pipelines with Azure DevOps, Jenkins, or similar tools.
  Data Visualization:
• Proficiency in Power BI for creating dashboards and reports.
• Experience with DAX and Power Query for advanced analytics.
  Data Integration and APIs:
• Experience with REST and SOAP APIs for data integration.
• Familiarity with messaging systems like Azure Event Hubs or Azure Service Bus.

General Skills:

• Strong problem-solving and analytical skills.
• Excellent communication and documentation skills. Ability to work in Agile/Scrum teams.

Preferred Skills:

• Experience with Azure Databricks for advanced analytics.
• Knowledge of machine learning workflows and integration with data engineering pipelines.
• Familiarity with streaming technologies like Azure Stream Analytics or Kafka.
• Understanding of data governance frameworks (e.g., Azure Purview).
• Hands on with C#, MVC .Net.

Education and Experience:

• Bachelor’s or master’s degree in computer science, Information Systems, or a related field.
• 3+ years of experience as a Data Engineer or in a similar role.

*******************************************************************************************************************************************