Lead Data Engineer

  • Manchester, UK
  • £100k
  • Permanent
  • Technology

Job description

Lead Data Engineer | £95,000 - £105,000 | Manchester or Warrington (Hybrid)

 

We're looking for a Lead Data Engineer to take ownership of the technical direction and continued development of a growing data platform.

A major part of the role will be leading the technical delivery of a large-scale data warehouse migration, moving an existing on-premise SQL Server environment towards a modern AWS-based lakehouse architecture.

You'll play a key role in shaping the platform's architecture, establishing engineering standards and driving improvements across data processing, streaming and overall platform performance.

This is a genuinely hands-on technical leadership position. You'll have the autonomy to make architectural decisions, challenge existing approaches and guide the wider engineering team, while remaining close to the code and day-to-day technical delivery.

 

What You'll Be Doing

 

  • Leading the technical delivery of a major data warehouse migration from on-premise SQL Server to an AWS-based lakehouse.
  • Defining the architecture, engineering standards and design patterns for the wider data platform.
  • Designing and developing scalable batch and real-time data processing solutions.
  • Building and improving data pipelines, ensuring they are reliable, efficient and maintainable.
  • Developing real-time streaming capabilities to significantly reduce data processing and reporting latency.
  • Establishing best practices around data modelling, data quality, validation and platform observability.
  • Taking ownership of infrastructure-as-code and CI/CD standards across the platform.

 

What We're Looking For

 

  • Significant experience in Data Engineering, with a proven track record of leading large-scale data platform or data warehouse migrations.
  • Deep, hands-on experience with AWS data services, particularly Glue, Lambda, S3, Redshift and EMR.
  • Advanced Python, PySpark and SQL skills.
  • Proven experience building and operating real-time streaming architectures using Apache Kafka or Amazon MSK.
  • Strong understanding of data warehousing, data modelling and modern lakehouse architecture.
  • Experience with open table formats such as Apache Iceberg or Delta Lake.
  • Experience with infrastructure-as-code, ideally Terraform, alongside CI/CD and pipeline orchestration.

 

If interested, please apply or reach out directly for a confidential chat!

Day-to-day

Develop and maintain software applications to meet client requirements.
Collaborate with cross-functional teams to design scalable solutions.
Troubleshoot and resolve technical issues promptly.
Participate in code reviews to ensure code quality and standards.
Document software functionality and update technical manuals.
Monitor system performance and implement improvements.
Stay updated with emerging technologies and integrate them as needed.
Provide technical support and training to end-users.

Cameron Whitwam
Share Job

Apply today

Optional description for this form can help users understand the purpose of this section.

Similar Jobs