Ayan Infotech are looking for a Data Engineer for a long term contract job opportunity in Melbourne with one of our private clients.

All applicants must have full unrestricted work rights in Australia.

Title: Data Engineer
Location: Melbourne
Type: Contract (6 months +)

Required Skills and Experience:
As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data infrastructure and pipelines. You will work closely with cross-functional teams to understand business requirements, optimize data workflows, and ensure the reliability and performance of our data systems.
Key Responsibilities:

• Design, build, and optimize data processing pipelines using Apache Spark, Python, Scala.
• Develop and maintain data ingestion and extraction processes, including streaming data pipelines using Kafka and batch processing workflows.
• Implement performance tuning techniques to optimize data processing and query performance, ensuring scalability and efficiency.
• Collaborate with DevOps teams to deploy and manage data infrastructure on NetApp S3 (very similar to AWS S3), Kubernetes.
• Containerize data applications using Docker and orchestrate deployment using Kubernetes for scalability and reliability.
• Develop and maintain unit tests using frameworks like pytest, junit, to ensure the quality and reliability of data pipelines.
• Implement and adhere to best practices for data governance, security, and compliance.
• Utilize Behavior-Driven Development (BDD) tools like Cucumber and Lettuce to write and execute test scenarios for data workflows.
• Stay up to date with emerging technologies and industry trends and evaluate their potential impact on our data infrastructure and processes.

Requirements:

• Proven experience in designing and building scalable data pipelines using Apache Spark, Python, and Scala.
• Strong understanding of data warehousing concepts, ETL processes, and data modelling techniques.
• Experience with performance tuning and optimization of Spark jobs and SQL queries.
• Familiarity with stream processing frameworks like Kafka and messaging systems.
• Hands-on experience with containerization and orchestration tools like Docker and Kubernetes.
• Experience with unit testing frameworks like pytest, junit, and BDD tools like Cucumber and Lettuce.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment

Contact: 02 9412 4178 for more details.

www.ayaninfotech.com