Senior Data Engineer
- القاهرة
- دائم
- دوام كامل
- Collaborate with CCH architects across different IT (DTPS) teams and with cross-functional teams to gather requirements, define technical specifications, and provide architectural guidance for Azure projects. Contribute and align on the strategy, design and implementation of the end2end data journey to expose analytics insights for consumption by different reporting layers.
- Ability to load data from Azure Data Sources (Azure SQL Databricks Warehouse/Synapse), visualize data, setup data refresh, analyze and write queries in SQL.
- Develop and implement data governance and data quality frameworks to ensure the integrity and accuracy of data;
- Design, develop, implement and optimize data pipelines for scalability, performance, security and cost efficiency. Optimize data storage, retrieval, and processing techniques to achieve high performance and cost efficiency.
- Implement security and access control measures to protect sensitive data within Azure environments.
- Perform data profiling, analysis, and validation to identify and address data quality issues.
- Develop and maintain documentation for data solutions, including data models, data flows, and technical specifications.
- Stay up to date with the latest Azure data services, tools, and best practices, and provide recommendations for their adoption.
- Lead and guide Azure developers and data engineers from vendors, providing guidance and support in Azure data engineering techniques and methodologies and ensuring development as per CCH standards and guidelines.
- Ability to understand business requirements and convert into clearly documented technical requirements (dimensions, KBIs, data structure, data granularity, target users), following the agreed guidelines, standards and rules.
- Collaborate with DevOps teams to implement continuous integration and delivery pipelines for Azure-based applications.
- Collaborate with and support Power BI developers to test different approaches for data connections and data transfers to Power BI, to identify the optimal approach per use case considering data volume, Power BI performance, as well as data quality and security considerations.
- Deep hands-on experience with MS Azure Data Factory, MS Azure Data Lake Storage, SQL Database (T-SQL), Azure Databricks, Proficiency in SQL and data manipulation languages for data extraction, transformation, and loading.
- Deep hands-on experience in Python.
- Experience with data visualization tools and techniques for creating meaningful data insights (e.g., Power BI, PowerApps) is appreciated.
- Experience with Data Science techniques, such as feature engineering and modeling is appreciated.
- Experience in performance optimization techniques in Azure technologies for PowerBI reporting consumption is a plus.
- Experience with big data architectures and large data volumes technologies (partitioned tables with billions of records, files with hundreds of GBs). Familiarity with big data technologies such as Hadoop, Spark, and Hive.
- Strong hands-on understanding of DevSecOps concepts and tools. Expertise in practical implementation of DevSecOps, in particular Azure DevOps toolkit and automated testing capabilities.
- Strong understanding of DevOps principles and experience with CI/CD pipelines, version control systems (e.g. Git), and automated deployment tools.
- Design and implement data pipelines and ETL processes, ensuring data quality and availability for machine learning models, while possessing a foundational understanding of machine learning concepts and techniques.
- Develop and maintain scalable machine learning pipelines and infrastructure, ensuring seamless integration and deployment of models in production environments (MLOps).
- Bachelor's degree in computer science, Engineering, or a related field (or equivalent practical experience).
- 5+ years of experience in data engineering, data science or related field, with a focus on Azure data platforms.
- Knowledge of SAP BW (extractors, transformations, queries, data export) would be appreciated.
- Understanding of Agile methodology and SAFe Agile is appreciated.
- Strong communication and interpersonal skills, with the ability to collaborate effectively with cross-functional teams, stakeholders and vendors.
- Azure certification, such as Azure Data Engineer Associate, is a plus.