We are looking for a highly skilled Senior Data Engineer with 7+ years of experience to join our team in Sydney. The role requires expertise in AWS services (EMR, S3, Athena, SQL) and ETL development using Spark and Scala. The ideal candidate will have experience in job scheduling (Autosys), debugging, and optimizing ETL pipelines. Prior exposure to ETL job migration and framework builds is preferred. This is a great opportunity to work on large-scale data projects, collaborating with multiple teams to deliver high-quality, scalable solutions.
Key Responsibilities:
- Execute and monitor ETL jobs in AWS from Day 1, ensuring smooth data processing.
- Reconcile job runs over multiple iterations to validate data integrity and accuracy.
- Identify, debug, and resolve issues within data pipelines, ensuring optimal performance.
- Collaborate with multiple capability teams to troubleshoot and optimize workflows.
- Ensure seamless knowledge transfer to offshore teams by documenting processes and best practices.
- Work on framework enhancements and process improvements for better efficiency.
- Contribute to ETL job migration projects, ensuring a smooth transition of data workflows.
Qualifications & Skills:
- 7+ years of experience in Data Engineering.
- Strong expertise in AWS services (EMR, S3, Athena, SQL).
- Hands-on experience with ETL development using Spark and Scala.
- Knowledge of job scheduling tools like Autosys.
- Proficiency in ETL lifecycle management – running, debugging, testing/reconciling, and deploying jobs in AWS.
- Prior experience with ETL job migration and framework builds (preferred).
- Strong analytical and problem-solving skills to resolve complex data challenges.
- Excellent communication and collaboration skills to engage with cross-functional teams.
This is an exciting opportunity to work on large-scale data projects in a fast-paced and innovative environment. If you are an experienced Senior Data Engineer with a passion for AWS-based data solutions, problem-solving, and optimizing ETL processes, we want to hear from you! Join our team in Sydney and contribute to building cutting-edge data solutions that drive business success.