AYAN InfoTech is looking for Data Engineer with (Apache Spark, Python, Scala) to join an exciting project based in Melbourne. The role offers you the opportunity to contribute towards an extremely well structured and mature environment, working on sophisticated enhancement projects.

Role: Data Engineer with Apache Spark, Python, Scala
Location: Melbourne
Contract Duration: 6 Months with high possible extensions
Experience: 5+ Years

Core Skills & Capabilities required:

• Design, build, and optimize data processing pipelines using Apache Spark, Python, Scala.
• Develop and maintain data ingestion and extraction processes, including streaming data pipelines using Kafka and batch processing workflows.
• Implement performance tuning techniques to optimize data processing and query performance, ensuring scalability and efficiency.
• Collaborate with DevOps teams to deploy and manage data infrastructure on NetApp S3 (very similar to AWS S3), Kubernetes.
• Containerize data applications using Docker and orchestrate deployment using Kubernetes for scalability and reliability.
• Develop and maintain unit tests using frameworks like pytest, junit, to ensure the quality and reliability of data pipelines.
• Implement and adhere to best practices for data governance, security, and compliance.
• Utilize Behavior-Driven Development (BDD) tools like Cucumber and Lettuce to write and execute test scenarios for data workflows.
• Stay up to date with emerging technologies and industry trends and evaluate their potential impact on our data infrastructure and processes.
• Proven experience in designing and building scalable data pipelines using Apache Spark, Python, and Scala.
• Strong understanding of data warehousing concepts, ETL processes, and data modelling techniques.
• Experience with performance tuning and optimization of Spark jobs and SQL queries.
• Familiarity with stream processing frameworks like Kafka and messaging systems.
• Hands-on experience with containerization and orchestration tools like Docker and Kubernetes.
• Experience with unit testing frameworks like pytest, junit, and BDD tools like Cucumber and Lettuce.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment

Contact: 61-(02) 7207 6926 for more details.

Please note we will be able to contact only shortlisted candidates for this role. We thank you in advance for your interest.