A senior data engineer designs and leads the implementation of data flows to connect operational systems, data for analytics and business intelligence (BI) systems.
At this role level, you will:
recognise opportunities to reuse existing data flows
lead the build of data streaming systems
optimise the code to ensure processes perform optimally
lead work on database management
Work Activity Components
| Title | Details |
|---|---|
| Data pipelines and stores (Level 4)(DENG) | Creates and maintains data pipelines to connect data within and between data stores, applications and organisations — engineering solutions may be on-premise, cloud-based or hybrid. |
| Coaching (Level 4) | Concepts, methods and techniques for providing coaching in subject specialisms to individuals or groups (e.g. GROW model). |
Technical Skills
| Title | Details | Depth |
|---|---|---|
| Database Software | Software that enables the user to capture, create, populate and manipulate data structures and where appropriate unstructured data. | Proficient in |
| Business Environment | The business environment relating to own sphere of work (own organisation and/or closely associated organisations, such as customers, suppliers, partners and competitors), in particular those aspects of the business that the specialism is to support (i.e. localised organisational awareness from a technical perspective). | Familiar with |
| Cloud/Virtualisation | The principles and application of cloud/ virtualisation (including ownership, responsibilities and security implications). Use of tools and systems to manage virtualised environments. | Proficient in |
Professional Development Activity (PDA)
| Title | Details | PDA Group |
|---|---|---|
| Deputising | Standing in for supervisor or manager on a temporary basis during periods of absence. | Broadening Activities |
| Job Shadowing and Special Assignments | Undertaking temporary periods or secondments in other roles, particularly those that offer a new perspective on own function or exposure to other environments and cultures. | Broadening Activities |
| Research Assignments | Exploring a topic which is not part of own normal responsibilities and presenting findings to colleagues and/or management | Increasing Knowledge |
| Gaining Knowledge of Activities of Employing Organisation | Developing an understanding of the potentially diverse range of activities (service, governance, administrative, regulatory, commercial, charitable, industrial, etc.) undertaken by the employing organisation. | Increasing Knowledge |
| Gaining Knowledge of IT Concepts and Techniques | Undertaking study, learning and, where possible, practice in IT concepts and techniques external to own function. | Increasing Knowledge |
| Participation in Professional Body Affairs | Taking an active part in professional body affairs at branch, specialist group, committee or board level. | Participation in Professional Activities |
| Negotiating and Influencing | Undertaking learning and practice of negotiating with and influencing others. | Developing Professional Skills |
Qualification Components
| Title | Awarding Bodies |
|---|---|
| FEDIP Senior Practitioner | The Federation for Informatics Professionals |
Additional Frameworks
National Competency Framework for Data Professionals in Health and Care
Behaviours
| Title | Details |
|---|---|
| Delivering outcomes (B1.1) (Level Three) | You initiate work with others in your team and beyond in order to bring about a generally agreed outcome. |
| Communicating within a hierarchy (B1.2) (Level Three) | You are able to communicate to those in authority when their requests are in conflict with each other. |
| Generating consensus (B1.3) (Level Three) | You are generally able to quickly gather a consensus of opinion to support your ideas. |
| Logical arguments (B1.4) (Level Three) | You are able to construct a clearly predicated argument with logically consistent conclusions. |
| Negotiation (B1.5) (Level Three) | You are able to negotiate exchanges over multiple poles of interest in order to achieve a specific result. |
| Generating support (B1.6) (Level Three) | You are always able to gather help with minimal effort when faced with a tricky problem. |
| Influence (B1.7) (Level Three) | You are able to apply influence even when faced with being influenced by a number of peers. |
| Equality (B2.1) (Level Three) | You make extra efforts to ensure that, where the voices of certain groups are not being heard, you take the time to hear them. |
| Challenging discrimination (B2.2) (Level Three) | You articulate, when prompted, the fact that discrimination of any kind will not be tolerated by your organisation and signpost issues to senior management for immediate resolution. |
| NHS Constitution (B2.3) (Level Three) | You have a detailed understanding of following the behaviours and values listed in the NHS Constitution. |
| Supporting others (B2.4) (Level Three) | You are supportive of others and take time to ensure they are okay. Whilst ensuring everyone has someone to talk to, you ensure that you are able to talk to someone when you need to as well. |
| Open environment (B2.5) (Level Three) | You help to model an open environment by facilitating sessions for underrepresented and marginalised individuals to share their lived experiences with you and your colleagues, asking questions and escalating concerns to your line manager. |
| Written communication (B3.1) (Level Three) | You are able to convey complex written ideas, citing a variety of sources, in a clear and concise manner. |
| Discussing complex ideas (B3.2) (Level Three) | You are able to discuss complex ideas in a clear and accessible manner to a non-technical audience. |
| Delivering complex ideas (B3.3) (Level Three) | You deliver complex ideas and insights to a diverse range of audiences. |
| Understanding new ideas (B3.4) (Level Three) | You are able to understand new and complex ideas from multiple sources in order to convey the information concisely. |
| Reading audiences (B3.5) (Level Three) | You are able to assess how comfortable someone is in understanding your information and naturally cover the information they are unclear on in a new and accessible manner. |
| Problem sharing (B4.1) (Level Three) | You regularly work on problems with others outside of your immediate work area while respecting pre-agreed boundaries. |
| Seeking opinions (B4.2) (Level Three) | You seek out the opinions of a number of others to create a rich solution to a problem. |
| Sharing best practice (B4.3) (Level Three) | You collate and offer best practice in your area and openly share your findings. |
| Embedding best practice (B4.4) (Level Three) | You seek out exemplars of "what good looks like" and embed them within your area. |
| Patient impact (B5.1) (Level Three) | You seek out ways of making your actions better for patients. |
| Understanding the customer (B5.2) (Level Three) | You take time to consult with the client, offering a range of solutions. |
| Customer service (B5.3) (Level Three) | You spend time on complex problems to understand the customer's perspective and solutions that would add value. |
| Customer solutions (B5.4) (Level Three) | You seek out new and innovative solutions to proactively inform customers of efficiencies in your provision. |
Leadership
| Title | Details |
|---|---|
| Empathy and understanding (Level Three) | You are always available to your team and consistently act with care, empathy and compassion. |
| Pressure (Level Three) | You are able to explain to those around you why organisational and political pressures make it seem like senior managers are pulling in a different direction sometimes. |
| EDI (Level Three) | You actively engage in your organisation's EDI networks and take care to appreciate the lived experiences of those people different to you. |
| Team support (Level Three) | You back your team consistently when questioned in public. |
| Positivity (Level Three) | You are able to act effectively and fairly when under pressure whilst being aware of the dangers of favouritism on morale. |
| Innovation (Level Three) | You seek out innovations that will support your team's work. |
| Safe to fail (Level Three) | You will promote your team's successes publicly and examine failures privately, taking responsibiity for those failures in public. |
| Fairness (Level Three) | You understand the preferences and strengths of individuals within your team and can utilise them effectively whilst developing their skills in other areas. |
| Opportunities (Level Three) | You empower the team to follow through on new ideas or processes, providing a safe environment in which to try new things. |
| Goals (Level Three) | You set clear goals and articulate achievable expectations. |
| Performance (Level Three) | You offer support to your team to exceed expectations. |
| Motivation (Level Three) | You consistently celebrate success but also promote learning through failure in safe environments. |
| Expectations (Level Three) | You are able to articulate disappointment when the efforts of others fall below expectations. |
| Succession planning (Level Three) | You are able to readily identify those in your team who have the opportunity to excel at their level and beyond and use this knowledge to begin succession planning. |
| Managing expenditure (Level Three) | You are able to manage your expenditure and ensure you have no quarterly or annual overspend. |
| Budget control (Level Three) | You are able to plan and manage your budgets ensuring that you have the enough money allocated to the correct area. |
| Forecasting (Level Three) | You are able to produce accurate forecasts based on current expenditure and foreseen developments within your team. |
| Business cases (Level Three) | You are able to highlight the need for a business case in new areas or work and identify options for appraisal. |
| Headcount (Level Three) | You are able to accurately plan workload to headcount, identifying how much is required in each area. |
| Recruitment (Level Three) | You take time to build your recruitment processes around professional standards whilst ensuring that the process remains open and transparent for all. |
| Supporting ambition (Level Three) | You understand the backgrounds and motivations for all members of your team and ensure that all efforts are made to support their ambitions whilst aligning them to the needs of the organisation. |
| Training opportunities (Level Three) | You provide ample opportunity for training for all members of your team whether that is from internal colleagues, external organisations or specialist providers. |
| Professional development (Level Three) | You ensure that all staff are able to focus a portion of their work time on CPD and ensure that their Personal Development Plans are regularly reviewed, updated and constantly worked on. |
| Managing external pressures (Level Three) | You ensure that time is set aside for the team to focus on each aspect of their role, wherever possible, free from distraction and interruption, protecting them, where possible, from the pressures of other managers outside the department, ensuring their well-being is protected from external pressures. |
Data Skills
| Title | Details |
|---|---|
| Stakeholder management (DEC1.2) (Level Three) | You manage stakeholder expectations. |
| Positive communications (Level Three) | You are flexible and capable of proactive and reactive communication. |
| Facilitation (Level Three) | You know how to facilitate difficult discussions within the team or with diverse senior stakeholders. |
| Data profiling (Level Three) | You develop advanced expertise in data profiling, applications, systems, platforms and reporting tools and pro-actively share your knowledge with others. |
| Data visualisation (Data Engineer) (Level Three) | You convert the results of source system analysis into insight to support the business. |
| Tools and techniques (Level Three) | Maintain knowledge of the breadth of techniques available for exploring, manipulating and creating new data sources through linking or matching multiple datasets. |
| Data management (Level Three) | Develop and maintain knowledge of database and data warehouse concepts, design principles and technologies to implement database / data warehouse designs that support demands of data analytics and improve the quality and flow of data used in organisational business intelligence tools. |
| Data requirements (Data Engineer) (Level Three) | Investigate data requirements where there is some complexity and ambiguity. |
| Data transformation (Data Engineer) (Level Three) | Provide advice on the transformation of data from one format or medium to another. |
| Data modelling (Data Engineering) (Level Three) | You understand the concepts and principles of data modelling and can produce relevant data models. |
| Reverse engineering (Level Three) | You know how to work across organisational boundaries and recognise opportunities to align with or re-use the data models in different organisations |
| Data integration (Level Three) | You establish sustainable enterprise-scale data integration procedures and ensure the team adheres to them. |
| Data services (Level Three) | You build appropriately skilled teams and manage resources to ensure data services work effectively at enterprise level. |
| Data engineering best practice (Level Three) | You provide advice and guidance to others using the data structures and associated components. |
| Ingestion (Level Three) | You can establish complex enterprise-scale data ingestion processes using standard development techniques. |
| Modernisation tools (Level Three) | You build appropriately skilled teams to support current and proposed data pipelines. |
| Information governance (DEC4.3) Level Three) | You can manage people and technical resources to ensure that data services work effectively at scale. |
| Data standards and architectures (Level Three) | You oversee the operational implementation of new technologies and contribute to the modernisation of the business data architecture. |
| Design (Level Three) | You ensure business processes build-in Information. |
| Data governance (Data Engineer) (Level Three) | Governance requirements and develop ethical and effective data sharing agreements for data sources. |
| Quality assurance (Data Engineer) (Level Three) | Support processes for complex data quality checking and remediation. |
| Policies and standards (Level Three) | Contribute to organisational policies, standards, and guidelines for data engineering. |
| Metadata repositories (Level Three) | You create metadata, design appropriate metadata repositories and manage changes to existing metadata repositories. |
| Metadata tools (Level Three) | You understand a range of tools for storing and working with metadata. |
| Metadata best practice (Level Three) | You enable the team by ensuring they have access to relevant tools and the appropriate levels of training. |
| Metadata processes (Level Three) | You review metadata processes and their documentation to ensure their efficiency and accuracy. |
| Programming (Data Engineer) (Level Three) | You utilise a variety of tools to design, code, test, correct and document moderate-to-complex programs and scripts from agreed specifications and subsequent iterations. |
| Development standards (Data Engineer) (Level Three) | You advise colleagues on industry standards and monitor adherance to local standards. |
| Testing (Level Three) | You collaborate with others to review specifications where appropriate and define test conditions and procedures. |
| Performance analysis (Level Three) | You analyse test and performance analysis results and identify and act upon any risks, issues or opportunities. |
| Development lifecycle (Level Three) | You direct code reviews and promote the application of a full range of refactoring techniques to enhance the lifespan of the code library. |
| Technologies (Level Three) | You apply detailed knowledge of the techniques and concepts required for the role and maintain currency of your own skills. |
| Emerging trends (Level Two) | You understand how these fit into the wider technical landscape. |
| Innovation (DEC7.3) (Level Three) | You research and identify areas for impactful innovation in data tools and techniques. |
| Stakeholder engagement (DEC7.4) (Level Three) | You collaborate with other technical teams to ensure seamless service to the organisation. |
| Geographical Data Mapping (Level Two) | You understand different geographies and how they can be displayed using point mapping, density mapping, chloropleth, isoline maps etc. You understand how geographical boundaries relate to eachother. |
| Data Modelling (Data) (Level Two) | You can express logical and physical data models to define how a model will be built. You understand the different data model infrastructures and the limitations of each. |
| Information Governance (Data) (Level Two) | You know the key data protection principles. You understand when data can be accessed and shared and know who to approach for advice/approval. You understand the impact of small numbers on identifiability of data. |
| Longitudinal Analysis (NCF Data) (Level One) | You can identify when longitudinal analysis is suitable and know the key techniques for analysis. |
| Technological Specialisms (R, Python, SQL, Tableau etc.) (Level Two) | You utilise associated modules and add-ins to perform complex manipulation and visualisation, data linkage and data quality. You can code to a standard to conduct work independently. |
| Data Automation (Level Two) | You can distinguish between data analytics automation and data flow automation. You can link to and produce data flow maps that show where automation would be beneficial. You can use tools e.g. Power Query to automate data processing tasks. |
| Non-technical audiences (Level Three) | You understand the needs of technical and business stakeholders and interpret them into technical requirements which can be implemented. |
Project Skills
| Title | Details |
|---|---|
| Business cases (WP1.1) (Level Three) | You assess business requirements with stakeholders and draft business cases for your area of expertise, highlighting benefits, risks and costs and ensuring issues such as sustainability are addressed where appropriate. |
| Scope (WP1.2) (Level Three) | You adopt rigorous scope, quality and change management practices and ensure they are adhered to throughout the team. |
| Reviews (WP1.3) (Level Three) | You contribute to project reviews, documenting and communicating where lessons are learned. |
| Advice and monitoring (WP2.1)(Level Three) | You advise on and monitor budgets for time, resources and other costs for data and digital project activities enforcing budget controls. |
| Complexity (Level Two) | You understand and can articulate when the complexity of a proposed project requires further professional management or support. |
| Scheduling (WP2.3)(Level Three) | You sequence the activities in data and digital projects logically, effectively and efficiently, incorporating any lessons learned from similar past projects. |
| Refinement (Level Two) | You refine the plan within your work area to take account of any authorised changes communicating actions, progress and results with project managers. |
| Resource identification (WP3.1)(Level Three) | You identify and budget for the resources required to fulfil the project plan's requirements. |
| Skill acquisition and management (WP3.2)(Level Three) | You plan for the recruitment of staff with additional required skill sets, liaising with HR and/or other providers to source skilled staff to fulfil project roles, onboard and manage them and their workloads. |
| Additional tools and resources (Level Three) | You cost and acquire, deploy and contract for the support of additional tools and resources such as hardware, software, training and data sources for the course of the project life cycle. |
| Resource allocation (Level Two) | You plan the allocation of existing resources to project work whilst effectively maintaining business as usual wherever feasible. |
| Project management (WP4.1)(Level Three) | You manage the allocation and co-ordination of project-related activities, collaborating with project managers to establish and embed key risk management and issue resolution processes. |
| Pilots and testing (WP4.2)(Level Three) | You lead on the scoping of testing and pilots, reporting key findings and re-assessing the expected benefits. |
| Implementation (WP4.3)(Level Three) | You engage with project beneficiaries to co-ordinate effective roll-out. |
| Communications (Level Two) | You communicate effectively with others, adapting your style and approach as and when required. |
| Business change (WP5.1)(Level Three) | You understand the business case for change and how data and digital can facilitate that, collaborating with project managers to define an achievable technological outcome which fosters new ways of working. |
| Assurance (WP5.2)(Level Three) | You positively influence stakeholders, building networks and alliances and lead the focus within data and digital on customer satisfaction. |
| Evaluation (WP5.3) (Level Three) | You ensure appropriate solutions are evaluated and viable alternatives are considered to deliver the intended business benefits. |
The Professional Body Responsible for this job family is AphA. This job role profile was created in collaboration with BCS, using Role Model Plus.