Data Automation Engineer
- بنها
- دائم
- دوام كامل
- Develop and implement automated data pipelines for ingesting, processing, and storing structured and unstructured data.
- Integrate data from various sources in both real-time and batch mechanisms and perform transformations to ensure consistency, accuracy, and reliability.
- Implement data quality checks and monitoring mechanisms to identify and address issues proactively.
- Write code and scripts for automation of data processes using languages such as Python, Java, or Scala.
- Leverage ETL (Extract, Transform, Load) tools and frameworks [Apache spark, AWS Glue] for efficient data processing.
- Collaborate with other intelligent data teams.
- Ensure seamless data flow for machine learning and analytics applications.
- Implement monitoring systems to track data pipeline performance and identify areas for optimization.
- Continuously optimize data processes for efficiency and scalability.
- Document data processes, system architecture, and codebase for knowledge sharing and future reference.
- Keep documentation up-to-date with any changes or enhancements to the data infrastructure.
- Implement and adhere to data security and compliance standards.
- Collaborate with security teams to ensure data protection and privacy.
- 1+ years of experience in data engineering, automation, or a related field.
- Strong understanding of data pipelines and data lifecycle management.
- Proficiency in scripting languages like Python, Shell, or similar.
- Experience with data automation tools (Airflow, Luigi, etc.) a plus.
- Excellent analytical and problem-solving skills.
- Strong communication and collaboration skills to work effectively across teams.
- Knowledge with cloud platforms (AWS, Azure, GCP) for data processing.