RDST3 Maintaining understandability and usability of raw data through external resources

Collection:

Title	Nexus data files from instruments
Description	These are data files captured straight from instruments. They contain measurements collected from instrument detectors. There is no typical size or number of a detector that an instrument has. For example, for STFC ISIS facility, the number of detector ranges from several thousands to a quarter of a million. The typical format of these files are raw or NeXus. The later is an international standard for neutron and synchrotron communities. The former is facility specific: many historic data files are in this format. Increasingly, NeXus format is being adopted as the standard format for instrument data.
Licensing	See the STFC Data Policy for the SCAPE project
Owner	STFC
Dataset Location	https://scapeweb.esc.rl.ac.uk/ (please get in touch with STFC for accessing the data)
Collection expert	Erica Yang (STFC)
Issues brainstorm	These are individual data files produced by the experiments. These files are readings of invididual experimental runs. They, themselves, do not have enough information to allow anybody to process them because, basically, they are neutron counts in the STFC ISIS facility case. They are raw data because it contains errors and noises that are needed to be removed before it can be analysed. Therefore, first of all, they have to be preserved alongside with the contextual information describing where it was produced (e.g. which instrument), when it was produced (which ISIS cycle), and what experiment it was produced for. All these information allow establishing the linkages between these raw files and relevant files generated at the same time while the files are being produced during an experiment. Other types of contextual information needed to be preserved include the software needed to process the files, the samples that are used to produce the files.
List of Issues	IS29 Characterisation and validation of very large data files IS30 Fixity capturing and checking of very large data files IS31 Semantic checking of very large data files IS32 Basic Migration of RAW to NeXus data IS33 Enhanced migration of RAW to NeXus data

Title	Scientific datasets relevant to STFC facilities
Description	There are three categories of scientific datasets: Raw data: data files captured from the instruments, they are typically in two formats: raw and NeXus. Nexus data files Catalogue data: each dataset from an experiment is associated with a set of catalogue entries, specified using the Core Scientific MetaData model (CSMD). The available sample catelogue data is XML under CSMD. ICAT data catalogue entries Processed data: data files captured through data reduction and analysis processes. Process datasets and the related context (workflow, software, provenance)
Licensing	See the STFC Data Policy for the SCAPE project
Owner	Research investigators, then, when the data becomes public, it will be owned by the public.
Dataset Location	Raw and catalogue data availability: https://scapeweb.esc.rl.ac.uk/ Processed data is not owned by STFC, so, not available
Collection expert	Erica Yang (STFC)
List of Issues	For 1st and 2nd SCAPE years: IS29 Characterisation and validation of very large data files IS30 Fixity capturing and checking of very large data files IS31 Semantic checking of very large data files IS32 Basic Migration of RAW to NeXus data IS33 Enhanced migration of RAW to NeXus data (Perhaps) for the 3rd SCAPE year and afterwards IS34 ISIS instrument website no longer applicable or available IS35 Mantid website or software no longer applicable or available IS36 Examine the long term value of the preserved datasets IS37 Preserving the verifiability and provenance of processed datasets

Title	ICAT Catalogue data
Description	Come in two types of formats: XMLs and database records
Licensing	See the STFC Data Policy for the SCAPE project
Owner	STFC
Dataset Location	https://scapeweb.esc.rl.ac.uk/ (for access, get in touch with STFC)
Collection expert	Erica Yang (STFC)
Issues brainstorm	these data is key to meaningful indexing and searching of data files within the data archive.
List of Issues	Not applicable to 1st and 2nd SCAPE years

Issues:

Issue 1

Title	IS34 ISIS instrument website no longer applicable or available
Detailed description	relevant description of the functioning of the instrument no longer available to help interpret data files
Scalability Challenge	What requirements are placed on the solution in terms of the SCAPE scales of scalability: content size, volume of content, complexity of content
Issue champion	Simon Lambert (STFC)
Other interested parties	Any other parties who are also interested in applying Issue Solutions to their Datasets. Identify the party with a link to their contact page on the SCAPE Sharepoint site, as well as identifying their institution in brackets. Eg: Schlarb Sven (ONB)
Possible Solution approaches	Brief brainstorm of possible approaches to solving the Issue. Each approach should be described in a single sentence as part of a bulleted list. Note that actual Solutions will be owned by the *Solution Provider* who should be a different person from the Issue Champion. Reaching a satisfactory conclusion for the Issue should be considered a team effort between these parties.
Datasets	Nexus data files, ICAT catalogue data
Solutions	Reference to the appropriate Solution page(s), by hyperlink

Evaluation Objectives	Not applicable for the 1st and 2nd SCAPE years
Actual evaluations	links to acutual evaluations of this Issue/Scenario

Issue 2

Title	Mantid website and/or software no longer applicable or available
Detailed description	1. Relevant description of the analysis software and the algorithms it implement no longer available. Impact: loss of the ability to use the Mantid software to properly process the raw data because, for example, the up-to-date manual is loss. 2. Data can no longer be analysed - i.e. becomes unusable, loss the capability of verifying the processed data
Issue champion	Simon Lambert (STFC)
Other interested parties	Any other parties who are also interested in applying Issue Solutions to their Datasets. Identify the party with a link to their contact page on the SCAPE Sharepoint site, as well as identifying their institution in brackets. Eg: Schlarb Sven (ONB)
Possible Solution approaches	1) This is a case where reasoning with a Preservation Network Model can help to find solutions. The dependencies indicate that either a local archive of the website or access to the UK Web Archiving Consortium archive of the STFC website would be able to substitute for the live site. 2)This is a case where reasoning with a Preservation Network Model can help to find solutions. The dependencies indicate that solutions could be provided by software on other platforms; specifications to reconstruct software.
Datasets	Nexus data files
Solutions	Reference to the appropriate Solution page(s), by hyperlink

Evaluation Objectives	Not applicable for the 1st and 2nd SCAPE years
Actual evaluations	links to acutual evaluations of this Issue/Scenario

Solutions:

Title	SO24 Use Preservation Network Model to record "deep" dependencies and to allow tracking over time
Detailed description	Preservation Network Model (PNM) is a methodology to record "deep" dependencies between digital artifacts and to allow tracking over time (this is a solution yet to develop)
Solution Champion	Simon Lambert (STFC)
Corresponding Issue(s)	IS35 Mantid website or software no longer applicable or available IS34 ISIS instrument website no longer applicable or available IS37 Preserving the verifiability and provenance of processed datasets Nexus data files from instruments
Evaluation	Not applicable to 1st and 2nd SCAPE years