DaSH Architecture
The Client Ask:
In May 2020, as the Covid-19 pandemic was gripping the UK, the Government initiated a new department hosted by the Cabinet Office, to start to process the vast amounts of data related to the outbreak. That department was to become the Joint Biosecurity Centre (JBC) NIAXO was selected to become the data platform delivery lead, not only to build tools and services for the JBC, but to consider the longer term data strategy. Given a blank sheet of paper as a design; with no tactical options in place, the team was asked to build from the ground up, with some very broad requirements, and a deadline of “yesterday”.
Delivering successful outcomes (NIAXO’s Solution):
With a pressing need to get data scientists and analysts from a range of government departments across the whole of the UK making sense of the data surrounding the pandemic, the key criteria was to get a system up and running as quickly as possible. However, the system also needed to be secure and extensible, due to the sensitive nature of the data, and the as-yet unknown user requirements and demands.
Using extensive experience of cloud-based platforms and data governance/management, the team implemented a data science platform for JBC analysts in a very short timescale and had it available for use within 2 weeks of our starting on the project. Key to this was the Data Governance Portal (a customised version of Jira) for managing data governance, data lifecycles, legal compliance and QA, and the Information Asset Register, a central data catalogue (built using CKAN), which manages over 300 datasets and more than 20,000 resources. Integration was achieved using an event-based model, leveraging serverless cloud functions to reduce costs and allow complex integrations.
Not only did users have a functional place to remotely access data science tooling, securely on the cloud, but also confidence that the data they were using had been governed correctly and was searchable through catalogue entries. All tools were developed and integrated as cloud-native, COTS or open-source in nature. RBAC and tightly controlled audit requirements were crucial to ensuring system security and accountability.
Throughout NIAXO’s time working alongside the JBC, additional developments were made to bring regular data feeds into the system automatically, with schema and AV checking as standard, as well as helping to innovate new solutions to data management and data science techniques. Also key to the success of the JBC was their ability to report their findings, through MI dashboards and reporting mechanisms, also managed as part of the governance lifecycle. One of NIAXO’s greatest successes was the deployment of an enterprise-scaled Qlik Sense cluster, which was greatly improved the JBC’s ability to understand, visualise and report its findings.
NIAXO’s expert team delivered a vast amount of infrastructure, code and functionality remotely during one of the most pressured development experiences of their careers. The platform has been converted to use Infrastructure-as-Code to allow the JBC to continue to develop and expand its ambitions, and at its peak supported well in excess of 450 users.
Value Added /NIAXO differentiators:
NIAXO has extensive experience in managing data in highly complex, and highly secure environments. Using existing design patterns and experience, NIAXO’s key differentiator was to be completely agile from the start, reacting to the needs of the JBC on an hourly basis, flexing the team where necessary. Additionally, NIAXO has key competencies in understanding data governance, lifecycle management and data protection. The aim was to build a system which was not only functionally performant, but also protected data owners and suppliers, data users, and the UK Government. This was built on a flexible and modular architecture that allowed for changes and additional requirements to be absorbed at pace without wholesale changes to the solution and the associated disruption to service.