Lead Data Engineer

Summary

A lead data engineer is responsible for the design and implementation of numerous complex data flows to connect operational systems, data for analytics and business intelligence (BI) systems.

At this role level, you will:

recognise and share opportunities to reuse existing data flows between teams

be responsible for the build of data-streaming systems

co-ordinate teams and set best practice and standards

apply knowledge of systems integration to your work

champion data engineering

Work Activity Components

Title Details
Data pipelines and stores (Level 5)(DENG) Plans and drives data engineering solution development ensuring that solutions balance functional and non-functional requirements.

Technical Skills

Title Details Depth
Business Environment The business environment relating to own sphere of work (own organisation and/or closely associated organisations, such as customers, suppliers, partners and competitors), in particular those aspects of the business that the specialism is to support (i.e. localised organisational awareness from a technical perspective). Proficient in
Cloud/Virtualisation The principles and application of cloud/ virtualisation (including ownership, responsibilities and security implications). Use of tools and systems to manage virtualised environments. Proficient in
Database Software Software that enables the user to capture, create, populate and manipulate data structures and where appropriate unstructured data. Proficient in

Training

Title Details
Data Management Data management concepts, methods, tools and techniques relating to the planning, development, implementation, administration and curation of data.
Technology Products for Future Use Technology products or solutions that are potentially of use to the organisation.
Coaching Concepts, methods and techniques for providing coaching in subject specialisms to individuals or groups (e.g. GROW model).

Professional Development Activity (PDA)

Title Details PDA Group
Deputising Standing in for supervisor or manager on a temporary basis during periods of absence. Broadening Activities
Gaining Knowledge of Activities of Employing Organisation Developing an understanding of the potentially diverse range of activities (service, governance, administrative, regulatory, commercial, charitable, industrial, etc.) undertaken by the employing organisation. Increasing Knowledge
Gaining Knowledge of IT Concepts and Techniques Undertaking study, learning and, where possible, practice in IT concepts and techniques external to own function. Increasing Knowledge
Job Shadowing and Special Assignments Undertaking temporary periods or secondments in other roles, particularly those that offer a new perspective on own function or exposure to other environments and cultures. Broadening Activities
Mentoring Acting as a mentor, advising those for whom there is no direct responsibility, on matters to do with their job role, career and professional development. Broadening Activities
Negotiating and Influencing Undertaking learning and practice of negotiating with and influencing others. Developing Professional Skills
Participation in Professional Body Affairs Taking an active part in professional body affairs at branch, specialist group, committee or board level. Participation in Professional Activities
Project Assignments Participating in a project team, working group or task force established to deliver a solution to a specific problem or issue - especially valuable if the group is inter-disciplinary. Broadening Activities
Research Assignments Exploring a topic which is not part of own normal responsibilities and presenting findings to colleagues and/or management Increasing Knowledge
Team Leadership Undertaking learning and practice of the skills required to lead teams, including motivation, direction, coaching, delegation, appraisal, counselling and developing others. Developing Professional Skills

Qualification Components

Title Awarding Bodies
FEDIP Advanced Practitioner The Federation for Informatics Professionals

Additional Frameworks

National Competency Framework for Data Professionals in Health and Care

Behaviours

Title Details
Delivering outcomes (B1.1) (Level Four) You are able to gather the skills of a diverse multi-disciplinary team in order to achieve an agreed outcome.
Communicating within a hierarchy (B1.2) (Level Four) You are able to challenge the use of hierarchical arguments where logic supports a different course of action and call out the use of emotional or coercive influence.
Generating consensus (B1.3) (Level Four) You are consistently able to gather a consensus of opinion to support your arguments and often know what people will support prior to discussion.
Logical arguments (B1.4) (Level Four) You are able to construct a clearly predicated argument with logically consistent conclusions whilst providing robust refutations of counterarguments.
Negotiation (B1.5) (Level Four) You are able to negotiate exchanges over multiple poles of interest in order to achieve a specific result even when those involved have hidden agendas, while allowing everyone to share multiple viewpoints.
Generating support (B1.6) (Level Four) Your team and colleagues will often go above and beyond to support your initiatives.
Influence (B1.7) (Level Four) Your opinion is often sought early by peers dealing with politically sensitive issues.
Equality (B2.1) (Level Four) You make extra efforts to ensure that, where the voices of certain groups are not being heard, you take the time to give them a voice.
Challenging discrimination (B2.2) (Level Four) You are able to engage with sensitive ED&I issues and deal with them with the utmost dignity, respect and fairness.  
NHS Constitution (B2.3) (Level Four) You promote the behaviours and values listed in the NHS Constitution.
Supporting others (B2.4) (Level Four) You view the wellbeing of you and those around you with high priority.  You take every step to ensure that people within your domain know that it's okay not to be okay.
Open environment (B2.5) (Level Four) You are an ally for underrepresented and marginalised groups and model an open environment by facilitating sessions for these individuals to share their lived experiences with you and your colleagues.
Challenging disrespect (B2.6) (Level Four) You support staff to understand the impact of disrespectful behaviour and support them in challenging it.
Written communication (B3.1) (Level Four) You are able to produce original written material that is accessible, referenced and publishable, including the production of literature reviews.
Discussing complex ideas (B3.2) (Level Four) You are able to engage in complex technical debates with other specialists whilst using accessible and accurate language.
Delivering complex ideas (B3.3) (Level Four) Your confidence in your expertise enables others to feel confident and at ease with your contribution.
Understanding new ideas (B3.4) (Level Four) You are able to design all insight into complex information in a way that is both accurate and concise.
Reading audiences (B3.5) (Level Four) You are able to read large audiences to assess how well they have understood a series of multi-disciplinary concepts.
Problem sharing (B4.1) (Level Four) You look to provide multi-disciplinary solutions for maximum adoption throughout the organisation, while respecting pre-agreed boundaries.
Seeking opinions (B4.2) (Level Four) You regularly create multi-disciplinary teams to address complex problems.
Sharing best practice (B4.3) (Level Four) You create an environment where the sharing of best practice is viewed as a central part of every review process.
Embedding best practice (B4.4) (Level Four) You ensure that all processes within your area are based on models of "what good looks like".
Patient impact (B5.1) (Level Four) You ensure work within your area is as efficient as possible and enables better health and care outcomes.
Understanding the customer (B5.2) (Level Four) You seek out opportunities to work collaboratively with customers to pre-empt requests.
Customer service (B5.3) (Level Four) You understand changes within health and care with a view offering solutions to foreseen requirements.
Customer solutions (B5.4) (Level Four) You apply new solutions to customer requirements in order to ensure maximum accuracy and efficiency.

Leadership

Title Details
Empathy and understanding (Level Four) You act with care, empathy and understanding and ensure that your team knows you are always available to them.
Pressure (Level Four) You are aware of the pressures faced by your senior managers, as well as those in your team, and are able to work collaboratively to ease them.
EDI (Level Four) You actively engage in your organisation's EDI networks to better understand and appreciate the lived experiences of people different to you and how you can create a working environment supportive to all.
Compassion (Level Four) You ensure that, whilst being a workplace, emotional issues are dealt with care and sensitivity.
Team support (Level Four) Your team feels supported and empowered to exceed their goals.
Positivity (Level Four) You set clear goals and expectations that are visible to your team and others in affiliated areas.
Innovation (Level Four) You build the importance of trying new things and failing in a controlled environment into your ways of working whilst always celebrating success.
Safe to fail (Level Four) You facilitate networking opportunities for your team, including those with external organisations.
Fairness (Level Four) You are regarded as fair by your team who consistently give you the best of their abilities.
Opportunities (Level Four) You identify opportunities for your team to learn from failure including fast fails and sandbox environments.
Goals (Level Four) You set challenging goals whilst empowering the team to explore a number of different solutions whilst keeping yourself available to support them.
Performance (Level Four) You regard sub standard work as a reflection of your ability as you had the opportunity to prevent it.
Motivation (Level Four) You use your knowledge of the motivations of your team to plan succession and support them in their ambitions beyond your team.
Expectations (Level Four) You are able to express the ramifications of below standard work in an honest and unemotional way.
Developing talent (Level Four) You take pride in the talent of your team and go to great lengths to develop their skills and innovations.
Succession planning (Level Three) You are able to readily identify those in your team who have the opportunity to excel at their level and beyond and use this knowledge to begin succession planning.
Managing expenditure (Level Four) You are able to agree and control expenditure required for the effective running of your team.
Budget control (Level Four) You agree and control budget allocations, highlighting anomalies in expenditure and make suggestions for the reallocations of funding.
Forecasting (Level Four) You are able to produce accurate forecasts based on current expenditure and foreseen developments within your division.
Business cases (Level Four) You deliver accurate and insightful business cases with appropriate and balanced options appraisals.
Headcount (Level Four) You are able to articulate the ramification of headcount on service provision to negotiate for alternative staffing levels.
Recruitment (Level Four) You support your team to recruit effectively whilst ensuring that fairness towards equality and diversity remains a priority throughout the process.
Supporting ambition (Level Four) You take the time to talk to all staff members within your remit and encourage your managers to support their ambitions, providing solutions where these diverge from perceived organisational drivers.
Training opportunities (Level Four) You ensure that your managers enable training on a regular basis within their unit and have ample access to training for their own development. You ensure that everyone understands delegation is a development opportunity that requires time and should not be seen as a shortcut to alleviating workloads.
Professional development (Level Four) You take an active interest in all staff members' PDPs within your area and ensure that development opportunities are seized upon.
Managing external pressures (Level Four) You ensure that time is set aside for the team to focus on each aspect of their role, wherever possible, free from distraction and interruption, protecting them, where possible, from the pressures of other managers outside the department, ensuring their well-being is protected from external pressures.

Data Skills

Title Details
Non-technical audiences (Level Three) You understand the needs of technical and business stakeholders and interpret them into technical requirements which can be implemented.
Stakeholder management (DEC1.2) (Level Four) You manage stakeholders’ expectations and facilitate discussions ainvolving high risk or complex issues or under constrained timescales.
Positive communications (DEC1.3) (Level Four) You can speak for and positively represent the technical community to large audiences inside and outside heathcare
Facilitation (Level Four) You can mediate between people and mend relationships, communicating with stakeholders at all levels.
Data profiling (Level Four) You have advanced expertise across a range of techniques or you may have a recognised deep expertise in a narrower range of specialisms.
Data visualisation (Data Engineer) (Level Four) You have advanced expertise in the relevant applications, systems and platforms in the organisation and share your knowledge with others.
Tools and techniques (Level Four) You exemplify best practice with methods and tools and guide others to a high standard.
Data management (DEC2.4) (Level Five) Derive an overall strategy of master data management that supports data exploitation for analysis, including linking/matching across diverse datasets.
Data requirements (Data Engineer) (Level Three) Investigate data requirements where there is some complexity and ambiguity.
Data transformation (Data Engineer) (Level Three) Provide advice on the transformation of data from one format or medium to another.
Data modelling (DEC3.1) (Level Four) You understand the concepts and principles of data modelling and can produce relevant data models.
Reverse engineering (DEC3.2) (Level Four) You know how to work across organisational boundaries and recognise opportunities to align with or re-use the data models in different organisations.
Data integration (Level Four) You establish sustainable enterprise-scale data integration procedures and ensure the team adheres to them.
Data services (Level Four) You manage resources to ensure data services work effectively at enterprise level.
Data engineering best practice (Level Four) You engage with the wider technical professions to ensure the data models and data engineering processes conform to best practice and data modelling principles.
Data iteration, review and maintenance (Level Four) You manage the iteration, review and maintenance of data requirements and data models.
Ingestion (Level Three) You can establish complex enterprise-scale data ingestion processes using standard development techniques.
Modernisation tools (Level Four) You pro-actively explore new tools and technologies, set direction for modernising data architecture, software and facilities and build business cases for technical investment that supports demands for onward uses.
Information governance (Data Engineering) (Level Four) You initiate novel data flows and their data sharing agreements ensuring Information Governance legislation is adhered to and data security standards are built-in at source. 
Data standards and architectures (Level Four) Monitors application of data standards and architectures including security and compliance.
Design (DEC4.5) (Level Four) Co-ordinate the application of data design and modelling techniques, to establish, modify or maintain data pipelines, data structures, datasets, and metadata for diverse datatypes.
Data governance (Data Engineer) (Level Three) Governance requirements and develop ethical and effective data sharing agreements for data sources.
Quality assurance (Data Engineer) (Level Three) Support processes for complex data quality checking and remediation.
Policies and standards (Level Four) You determine technical strategy, organisational policy, standards and guidelines and ensure alignment with enterprise goals and functional and non-functional requirements.
Metadata repositories (Level Four) You understand how metadata and their repositories support different areas of the business.
Metadata tools (Level Four) You promote and communicate the value of metadata repositories.
Metadata best practice (Level Four) You establish robust governance processes to ensure repositories are secure, up to date and conforming to best ethical practices.
Metadata processes (Level Three) You review metadata processes and their documentation to ensure their efficiency and accuracy.
Programming (Data Engineer) (Level Four) You set local standards for programming tools and techniques and nominate appropriate development methods.
Development standards (Data Engineer) (Level Four) You advise on the application of standards and methods and ensure compliance.
Testing (Level Four) You apply stringent test conditions, ensuring action is taken to counter all identified issues and risks.
Performance analysis (Level Four) You take technical responsibility for all stages and/or iterations in a development project, providing method-specific technical advice and guidance to project stakeholders.
Development lifecycle (Level Four) You take responsibility for the lifecycle of the whole code library sponsoring continuous code reviews and refactoring.
Technologies (Level Four) You predict and advise on future technologies that present opportunities for products or programmes.
Emerging trends (Level Four) You investigate emerging trends in data-related approaches, perform horizon scanning for the wider technical community and maintain the currency of your own skills.
Innovation (DEC7.3) (Level Three) You research and identify areas for impactful innovation in data tools and techniques.
Stakeholder engagement (DEC7.4) (Level Three) You collaborate with other technical teams to ensure seamless service to the organisation.
Geographical Data Mapping (Level Two) You understand different geographies and how they can be displayed using point mapping, density mapping, chloropleth, isoline maps etc. You understand how geographical boundaries relate to eachother.
Data Modelling (Level Three) You can build appropriate data models from physical data models and pick the most appropriate infrastructure. You understand data entities, attributes and specific modelling environments, e.g. Oracle, SQL Server, Hadoop etc.
Information Governance (Data) (Level Two) You know the key data protection principles. You understand when data can be accessed and shared and know who to approach for advice/approval. You understand the impact of small numbers on identifiability of data.
Longitudinal Analysis (Level Two) You can identify when longitudinal analysis is suitable and know the key techniques for analysis. You understand repeated measures designs and the limitations of standard statistical techniques.
Technological Specialisms (R, Python, SQL, Tableau etc.) (Level Three) You can produce complex data models and visualisations whilst ensuring accurate linkage and data quality. You use appropriately advanced coding and debugging skills to utilise and contribute to the open source community.
Data Automation (Level Three) You are able to link directly to source data using appropriate tools. You understand the inbuilt functionality of Microsoft and other products to directly link to SQL servers, Azure etc. You have knowledge of APIs and how they may benefit automation.

Project Skills

Title Details
Business cases (Level Four) You instigate business case development and work with project management colleagues to define project requirements, scope and overall time, quality and cost constraints.
Scope (Level Four) You define and engage stakeholders.
Reviews (Level Four) You advise on the coherence of programmes in data and digital to maximise the effectiveness of time and available skills within the business.
Quality assurance - Data and digital (Level Four) You lead quality assurance in data and digital, drawing on external expertise where necessary, learning lessons and sharing those with the wider data and digital community for their project work.
Advice and monitoring (Level Four) You monitor and manage the capacity of data and digital teams to meet current project plans, escalating any issues with skills, timeframes and other resources impacting upon project plans with colleagues in project management.
Complexity (Level Two) You understand and can articulate when the complexity of a proposed project requires further professional management or support.
Scheduling (WP2.3)(Level Three) You sequence the activities in data and digital projects logically, effectively and efficiently, incorporating any lessons learned from similar past projects.
Refinement (Level Two) You refine the plan within your work area to take account of any authorised changes communicating actions, progress and results with project managers.
Resource identification (Level Four) You ensure resources for projects are in place and optimised across any programmes, communicating their business value to non-data and digital colleagues and stakeholders.
Skill acquisition and management (WP3.2)(Level Three) You plan for the recruitment of staff with additional required skill sets, liaising with HR and/or other providers to source skilled staff to fulfil project roles, onboard and manage them and their workloads.
Additional tools and resources (Level Three) You cost and acquire, deploy and contract for the support of additional tools and resources such as hardware, software, training and data sources for the course of the project life cycle.
Resource allocation (Level Two) You plan the allocation of existing resources to project work whilst effectively maintaining business as usual wherever feasible.
Project management (Level Four) You ensure the team's project delivery activities have sufficient resources to co-exist with business as usual.
Pilots and testing (Level Four) You manage the risks and issues affecting data and digital roles in the project or programme.
Implementation (Level Four) You engage with project managers and stakeholders to map out all necessary resources and activities for effective and sustained implementation
Communications (Level Two) You communicate effectively with others, adapting your style and approach as and when required.
Business change (Level Four) You promote how data and digital can champion business change and identify further technological opportunities to bring about business benefits.
Assurance (Level Four) You provide assurance that the business benefits identified for a project can be realised, refining options for delivery and managing change control processes.
Evaluation (WP5.3) (Level Three) You ensure appropriate solutions are evaluated and viable alternatives are considered to deliver the intended business benefits.

The Professional Body Responsible for this job family is AphA. This job role profile was created in collaboration with BCS, using Role Model Plus.

Give Feedback

The Occupational Architecture Project is interactive and dynamic


If you would like to provide feedback on this job role, or the job families, please click the button below.

Give Feedback