Data Engineer

About the job

Key responsibilities

1. Working with colleagues to understand and implement requirements

1. Securing the data supply chain, understanding how data is ingested from different sources and combined / transformed into a single data set.

1. Understanding how to analyse, cleanse, join and transform data.

1. Implementing designed / specified solutions into the chosen platform

1. Working with colleagues to ensure that the on-prem/cloud infrastructure available is capable of meeting the solution requirements.

1. Planning, designing, and conducting tests of the implementations, correcting errors and retesting to achieve an acceptable result.

1. Appreciating how to manage the data including security, archiving, structure and storage.

Skill(s) required

Data Analysis Data Engineering Engineering Drawing Engineering Surveying MS SQL Server Python Python Libraries Statistical Modeling

Who can apply

Only those candidates can apply who

1. have minimum 5 years of experience

Other requirements

1. Degree level education in Mathematics, Scientific, Computing or Engineering discipline or equivalent experience

1. 6+ years of experience at various levels of Data Engineering roles including 3 years in technical lead role managing end to end solution.

1. Hands on Experience with Azure Databricks using PYSPARK with ability to do cluster capacity planning and workload optimization.