Summary
A data engineer delivers the designs set by more senior members of the data engineering community.
At this role level, you will:
implement data flows to connect operational systems, data for analytics and business intelligence (BI) systems
document source-to-target mappings
re-engineer manual data flows to enable scaling and repeatable use
support the build of data streaming systems
write ETL (extract, transform, load) scripts and code to ensure the ETL process performs optimally
develop business intelligence reports that can be reused
build accessible data for analysis
Work Activity Components
| Title | Details |
|---|---|
| Data pipelines and stores (Level 3)(DENG) | Designs and implements data pipelines and data stores to acquire and prepare data. Applies data engineering standards and tools to create and maintain data pipelines and data stores. |
Technical Skills
| Title | Details | Depth |
|---|---|---|
| Database Software | Software that enables the user to capture, create, populate and manipulate data structures and where appropriate unstructured data. | Familiar with |
| Business Environment | The business environment relating to own sphere of work (own organisation and/or closely associated organisations, such as customers, suppliers, partners and competitors), in particular those aspects of the business that the specialism is to support (i.e. localised organisational awareness from a technical perspective). | Aware of |
| Middleware | Software which forms part of the operating platform infrastructure. | Proficient in |
| Cloud/Virtualisation | The principles and application of cloud/ virtualisation (including ownership, responsibilities and security implications). Use of tools and systems to manage virtualised environments. | Familiar with |
Professional Development Activity (PDA)
| Title | Details | PDA Group |
|---|---|---|
| Deputising | Standing in for supervisor or manager on a temporary basis during periods of absence. | Broadening Activities |
| Job Shadowing and Special Assignments | Undertaking temporary periods or secondments in other roles, particularly those that offer a new perspective on own function or exposure to other environments and cultures. | Broadening Activities |
| Gaining Knowledge of Employing Organisation | Gaining basic knowledge of the employing organisation, its business, structure, culture, policies, products/services, operations and terminology. | Increasing Knowledge |
| Gaining Knowledge of the Technical Environment | Gaining knowledge of IT activities in the employing organisation. | Increasing Knowledge |
| Research Assignments | Exploring a topic which is not part of own normal responsibilities and presenting findings to colleagues and/or management | Increasing Knowledge |
| Involvement in Professional Body Activities | Attending meetings, seminars and workshops organised by professional body and reading published material, such as journals and web content. | Participation in Professional Activities |
| Communications | Undertaking learning and practice in oral and written communications, including report writing and presentation. | Developing Professional Skills |
Qualification Components
| Title | Awarding Bodies |
|---|---|
| FEDIP Practitioner | The Federation for Informatics Professionals |
Additional Frameworks
National Competency Framework for Data Professionals in Health and Care
Behaviours
| Title | Details |
|---|---|
| Delivering outcomes (B1.1) (Level Two) | You initiate work with others in your team in order to bring about a generally agreed outcome. |
| Communicating within a hierarchy (B1.2) (Level Two) | You are able to appeal to those in authority in order to facilitate influence. |
| Generating consensus (B1.3) (Level Two) | You understand how to steer the direction of activity by influencing the consensus of opinion. |
| Logical arguments (B1.4) (Level Two) | You understand how to structure a reasoned argument to influence the decisions of others. |
| Negotiation (B1.5) (Level Two) | You are able to negotiate simple exchanges in order to achieve a specific result. |
| Generating support (B1.6) (Level Two) | You understand how interpersonal skills such as kindness and compassion can increase people's desire to support you. |
| Influence (B1.7) (Level Two) | You are generally able to identify when you are being influenced and assess the situation on its own merits. |
| Equality (B2.1) (Level Two) | You consistently look to collaborate with colleagues who are representative of the protected characteristics in the Equality Act 2010 to work on any group and do not tolerate any implication that any group should be omitted from inclusion. |
| Challenging discrimination (B2.2) (Level Two) | You articulate, when prompted, the fact that discrimination of any kind will not be tolerated by your organisation and report what you know to your line manager. |
| NHS Constitution (B2.3) (Level Two) | You know the importance of following, and are familiar with, the behaviours and values listed in the NHS Constitution. |
| Supporting others (B2.4) (Level Two) | You are kind to yourself, supportive of those around you and let someone know if things become difficult. |
| Open environment (B2.5) (Level Two) | You have read your organisation's local and the NHS National Equality, Diversity and Inclusion policies and take time to actively listen to the lived experiences of underrepresented and marginalised groups, asking questions and escalating the concerns of your colleagues to your line manager. |
| Written communication (B3.1) (Level Two) | You are able to convey complex written ideas and insights in a clear and concise manner. |
| Discussing complex ideas (B3.2) (Level Two) | You are able to discuss complex ideas in a clear and concise manner. |
| Delivering complex ideas (B3.3) (Level Two) | You are able to appear confident when conveying complex ideas and insights. |
| Understanding new ideas (B3.4) (Level Two) | You are able to understand new and complex ideas when brought up in conversation. |
| Reading audiences (B3.5) (Level Two) | You consistently check to ensure other parties have understood the message from your communication. |
| Problem sharing (B4.1) (Level Two) | You regularly work on problems with colleagues in your team. |
| Seeking opinions (B4.2) (Level Two) | You take time to elicit the input of others to a problem. |
| Sharing best practice (B4.3) (Level Two) | You look to make successes part of your routine offering. |
| Embedding best practice (B4.4) (Level Two) | You scan the successes of the team in order to improve your work. |
| Patient impact (B5.1) (Level Two) | You understand the impact of your actions on patients. |
| Understanding the customer (B5.2) (Level Two) | You spend time with the customer to understand what will add value to their requirement. |
| Customer service (B5.3) (Level Two) | You use your knowledge and experience to offer alternative suggestions that would benefit the customer. |
| Customer solutions (B5.4) (Level Two) | You regularly try new techniques to provide greater efficiencies for the customer or outcomes for the patient. |
Data Skills
| Title | Details |
|---|---|
| Non-technical audiences (Level Two) | You can effectively communicate to and between technical and non-technical stakeholders and facilitate discussions within a multidisciplinary team, with some potentially difficult dynamics. |
| Stakeholder management (Level Two) | You can advocate for the team externally. |
| Positive communications (Level Two) | You know how to manage different perspectives. |
| Data profiling (Level Two) | You develop expertise in data profiling, locally used applications, systems, platforms and reporting tools and share your knowledge with others. |
| Data visualisation (Data Engineer) (Level Two) | You can apply a range of data visualisation practices and can advise on best practice and guide others to a high standard. |
| Tools and techniques (Level Two) | You can determine which tools and techniques to use to explore or solve a variety of business issues and communicate the results of analysis with impact to a range of audiences. |
| Data management (Level Two) | Recognise where external data sources can be integrated with own datasets. |
| Data transformation (Data Engineer) (Level Two) | Create new datasets through the manipulation of multiple data sources, including linking or matching data using techniques already established in the work area. |
| Data modelling (Data Engineering) (Level Two) | You understand the concepts and principles of data modelling and can produce relevant data models across multiple subject areas. |
| Reverse engineering (Level Two) | Working with senior colleagues you understand how to reverse-engineer data models from existing systems. |
| Data integration (Level Two) | You understand industry-recognised data modelling patterns and standards and when to apply them based upon a detailed understanding of requirements. |
| Data services (Level Two) | You design, build and test complex or large-scale data structures and associated components and liaise with colleagues to create data pipelines for services. |
| Ingestion (Level Two) | You can design, build, test, modify and maintain data pipelines and data stores creating complex or large-scale data products. |
| Modernisation tools (Level Two) | You understand how the data engineering standards, tools and technologies fit into the business data architecture. |
| Information governance (Data Engineering) (Level Two) | You understand the Information Governance requirements for the data you handle and can perform administrative tasks to provide accessibility, retrievability, security and protection of data |
| Quality assurance (Data Engineer) (Level Two) | Consideration of data quality issues and remediation. |
| Metadata repositories (Level Two) | You create metadata, design appropriate metadata repositories and manage changes to existing metadata repositories. |
| Metadata tools (Level Two) | You understand a range of tools for storing and working with metadata. |
| Metadata best practice (Level Two) | You provide oversight and advice to more inexperienced members of the team. |
| Metadata processes (Level One) | You work with metadata to complete tasks such as data and systems integration impact analysis. |
| Programming (Data Engineer) (Level Two) | You utilise standard tools to design, code, test, correct and document moderate-to-complex programs and scripts from agreed specifications and subsequent iterations. |
| Development standards (Data Engineer) (Level Two) | You collaborate with others to review specifications where appropriate and define test conditions and procedures. |
| Testing (Level Two) | You analyse and report test results. |
| Performance analysis (Level Two) | You perform program performance analysis reporting on a variety of appropriate metrics |
| Development lifecycle (Level Two) | You initiate code reviews and promote the use of refactoring techniques to enhance the lifespan of the code library. |
| Technologies (Level Two) | You apply detailed knowledge and understanding of the technical concepts required for the role. |
| Emerging trends (Level Two) | You understand how these fit into the wider technical landscape. |
| Innovation (DEC7.3) (Level Two) | You grasp the impact of applying emerging trends in data and analysis tools and techniques on the team. |
| Geographical Data Mapping (Level One) | You understand how geographical data can be displayed to show geographical features such as simple chloropleth mapping using appropriate tools. |
| Data Modelling (Data) (Level Two) | You can express logical and physical data models to define how a model will be built. You understand the different data model infrastructures and the limitations of each. |
| Information Governance (Data) (Level One) | You know the key data protection principles. You understand when data can be accessed and shared and know who in the organisation to approach for advice/approval. |
| Technological Specialisms (R, Python, SQL, Tableau etc.) (Level Two) | You utilise associated modules and add-ins to perform complex manipulation and visualisation, data linkage and data quality. You can code to a standard to conduct work independently. |
| Data Automation (Level Two) | You can distinguish between data analytics automation and data flow automation. You can link to and produce data flow maps that show where automation would be beneficial. You can use tools e.g. Power Query to automate data processing tasks. |
Project Skills
| Title | Details |
|---|---|
| Business cases (WP1.1) (Level Two) | You advise on the effort required and perceived risks and benefits during the development of business cases. |
| Scope (Level Two) | You ensure the team's work conforms to project scope, adjusting as necessary following any authorised changes, allocating work efficiently whilst maintaining quality standards within the allocated timescales. |
| Reviews (Level Two) | You report progress effectively using the project management framework in use demonstrating accountability for the team's output. |
| Advice and monitoring (Level Two) | You advise on data and digital project plans to ensure they thoroughly encompass all the activities and resources required to ensure a successful outcome and that the planned benefits can be realised. |
| Complexity (Level Two) | You understand and can articulate when the complexity of a proposed project requires further professional management or support. |
| Scheduling (Level Two) | You schedule project work appropriately for yourself and the team, ensuring business needs are met both within the project and in business as usual. |
| Refinement (Level Two) | You refine the plan within your work area to take account of any authorised changes communicating actions, progress and results with project managers. |
| Resource identification (Level Two) | You identify the resources required to fulfil the project plan's requirements. |
| Skill acquisition and management (Level Two) | You plan for the recruitment of staff with additional required skill sets, liaising with HR and/or other providers to source skilled staff to fulfil project roles. |
| Additional tools and resources (Level Two) | You plan for the acquisition, deployment and support of additional tools and resources such as hardware, software and data sources for the course of the project. |
| Resource allocation (Level Two) | You plan the allocation of existing resources to project work whilst effectively maintaining business as usual wherever feasible. |
| Project management (Level Two) | You identify and co-ordinate team project-related activities to conform to cost, time and quality limitations, reporting and, where approriate, managing any risks and issues locally. |
| Pilots and testing (Level Two) | You advise on or lead pilot and testing programs and report on progress, findings and lessons learned. |
| Implementation (Level Two) | You engage with project co-ordinators and change agents to facilitate the implementation of a project and nurture its sustainability. |
| Communications (Level Two) | You communicate effectively with others, adapting your style and approach as and when required. |
| Business change (Level Two) | You understand the business case for change and how your team can facilitate that change. |
| Assurance (Level Two) | You contribute to ideas generation and the evaluation of appropriate solutions which deliver the intended business benefits. |
| Evaluation (Level Two) | You are an advocate for the project and the benefits to be realised. |
The Professional Body Responsible for this job family is AphA. This job role profile was created in collaboration with BCS, using Role Model Plus.