Delivery Centric is seeking a Senior GCP Data Architect / Data Engineering Lead to design and deliver large-scale real-time and batch transaction data platforms on Google Cloud. The role combines solution architecture, hands-on data engineering, technical leadership and customer collaboration.
Key Responsibilities
- Lead technical architecture and solution design for large-scale transaction data platforms.
- Provide technical guidance on GCP architecture, data modelling, schema design and engineering best practices.
- Design and implement real-time and batch data ingestion and processing pipelines.
- Develop pipelines including Kafka to Spanner/BigQuery via Dataflow and GCS to Spanner/BigQuery.
- Independently develop, unit test and hand over production-ready data solutions.
- Perform pair programming and provide technical guidance to data engineering teams.
- Lead sprint planning, backlog management, scope definition, risks, issues and delivery timelines.
- Work as a technical workstream or overall project technical lead depending on project scale.
- Design solutions for migrating existing data platforms to GCP cloud-native services and for greenfield GCP implementations.
- Optimise BigQuery queries and data workloads for performance and scalability.
- Develop CI/CD pipelines for data workloads using Cloud Build, Artifact Registry and Terraform.
- Design orchestration workflows using Cloud Composer and Cloud Workflows.
- Support data governance and cataloguing using Dataplex and Data Catalog.
- Apply Generative AI technologies and integrations where appropriate within enterprise data solutions.
Must-Have Skills
- Strong hands-on experience with Google Cloud Platform data and analytics technologies.
- Experience designing large-scale transaction data platforms on GCP.
- Strong experience with GCP Dataflow, BigQuery and Cloud Spanner.
- Experience with real-time/event-driven data ingestion using Confluent Kafka and Pub/Sub.
- Experience with Cloud Composer and Cloud Workflows.
- Strong experience with Terraform and Cloud Build for data engineering workloads.
- Hands-on programming experience with Python and Java.
- Experience designing and implementing batch and event-driven data pipelines from sources including databases, files, APIs and streaming platforms.
- Experience with BigQuery performance optimisation.
- Experience with CI/CD, infrastructure automation and production data workload deployment.
- Strong understanding of data architecture, solution design and engineering best practices.
Good-to-Have Skills
- DBT or Dataform.
- Experience with Cloud Run and enterprise GCP services.
- Experience with Dataplex and Data Catalog.
- Experience modernising or migrating legacy data platforms to GCP.
- Experience with legacy technologies such as Hadoop, Spark, Cloudera/Hortonworks, HBase and Oozie.
- Experience with NoSQL databases such as HBase or MongoDB.
- Knowledge of Spanner, Bigtable, Cloud SQL and database migration approaches.
- Experience modernising legacy data-processing logic such as Hive, Teradata and Python into BigQuery SQL.
- Experience applying Generative AI technologies within enterprise environments.
- GCP Data Engineering certification.
Key Skills
GCP | Dataflow | BigQuery | Cloud Spanner | Pub/Sub | Confluent Kafka | Cloud Composer | Cloud Workflows | Terraform | Cloud Build | Python | Java | Real-Time Data | Batch Data | Transaction Data Platforms | Data Architecture | Data Engineering | Technical Leadership
How to Apply
Ready to take on a Senior GCP Data Architect / Data Engineering Lead role in Sydney/Melbourne with Delivery Centric? Apply now to join a team that delivers excellence and realises value for its clients.