The NERC Digital Gathering #DG23 Hackathon event, held following the conference has now finished. We adopted a ‘collaborative’ approach to this year’s hackathon, with two groups of experienced instructors helping to facilitate the delegates to use the software tools and computational environments on offer to further their research. Expert Network member Prof. Richard Reeve and team helped delegates learn about the FAIR Data Pipeline tools they have developed, with Expert Network member Dr Claire Harris providing delegates with worked example case studies of biological system modelling to show the tools in practice. The FAIR Data Pipeline (https://www.fairdatapipeline.org) is intended to enable tracking of provenance of FAIR (findable, accessible, interoperable and reusable) data used in modelling. Pipeline APIs written in C++, Java, Julia, Python and R can be called by modelling software for data ingestion. These interact with a local relational database storing metadata and the local filesystem, and are configured using a yaml file associated with the model run. Local files and metadata can be synchronised with a remote registry via a command line tool.
Also co-leading the Hackathon event were Matt Pritchard and colleagues from the STFC JASMIN team. JASMIN is NERC’s globally-unique data analysis facility, providing storage and compute facilities to enable data-intensive environmental science. Delegates were walked through migrating existing modelling approaches onto first the interactive shell, then the batch environments in JASMIN, plus the use of the interactive Jupyter Notebook environment. Also covered was how models can make use of and draw upon the extensive data holdings of the CEDA archives.
Overall the hackathon was a great success and highlighted the value of bringing a group of digitally-enabled scientists together with expert instructors to move forward their research techniques. We would like to see far more of this sort of offering in the future. Particular thanks go to the instructor teams.