Delivery Centric is seeking a seasoned Databricks Tech Lead to design, build, and scale data ingestion and processing frameworks. The role will lead a team of data engineers, drive software engineering best practices, and develop reusable, highly scalable pipelines to ingest and transform data from diverse enterprise sources into a Lakehouse platform.
Key Responsibilities
- Design and build metadata-driven, generic data ingestion frameworks supporting batch, streaming, and CDC to automate the onboarding of new data sources.
- Define architectural standards for Medallion architecture (Bronze, Silver, Gold) using Delta Lake.
- Optimise Spark jobs for performance, scalability, and cost efficiency across the Databricks platform.
- Implement and mature CI/CD pipelines using Git and automated testing, while enforcing DataOps best practices.
- Integrate data pipelines with Unity Catalog to support enterprise data governance and lineage.
Required Qualifications
- 8+ years of Data Engineering experience, including 3–4 years of hands-on technical leadership experience leading engineering teams or pods.
- Advanced proficiency in PySpark and Databricks SQL, with strong hands-on experience in Delta Lake and Unity Catalog.
- Proven experience building custom or highly configurable data ingestion frameworks for structured and unstructured data.
- Strong experience with AWS, Azure, or GCP cloud-native services such as S3 or ADLS.
- Deep understanding of data modelling, including star/snowflake schemas and SCD (Slowly Changing Dimensions), along with strong software engineering principles applied to data platforms.
- Excellent stakeholder management skills and the ability to translate complex technical architectures into business value.
Preferred Qualifications
- Experience with Kafka, dbt, or Delta Live Tables (DLT).
- Databricks Certified Data Engineer Associate or Professional certification.
How to Apply
Ready to take on a Databricks Tech Lead role in Sydney with Delivery Centric? Apply now to join a team that delivers excellence and realises value for its clients.