RDST2 Format Migration of (raw) Scientific Datasets

Collection:

Title	Nexus data files from instruments
Description	These are data files captured straight from instruments. They contain measurements collected from instrument detectors. There is no typical size or number of a detector that an instrument has. For example, for STFC ISIS facility, the number of detector ranges from several thousands to a quarter of a million. The typical format of these files are raw or NeXus. The later is an international standard for neutron and synchrotron communities. The former is facility specific: many historic data files are in this format. Increasingly, NeXus format is being adopted as the standard format for instrument data.
Licensing	See the STFC Data Policy for the SCAPE project
Owner	STFC
Dataset Location	https://scapeweb.esc.rl.ac.uk/ (please get in touch with STFC for accessing the data)
Collection expert	Erica Yang (STFC)
Issues brainstorm	These are individual data files produced by the experiments. These files are readings of invididual experimental runs. They, themselves, do not have enough information to allow anybody to process them because, basically, they are neutron counts in the STFC ISIS facility case. They are raw data because it contains errors and noises that are needed to be removed before it can be analysed. Therefore, first of all, they have to be preserved alongside with the contextual information describing where it was produced (e.g. which instrument), when it was produced (which ISIS cycle), and what experiment it was produced for. All these information allow establishing the linkages between these raw files and relevant files generated at the same time while the files are being produced during an experiment. Other types of contextual information needed to be preserved include the software needed to process the files, the samples that are used to produce the files.
List of Issues	IS29 Characterisation and validation of very large data files IS30 Fixity capturing and checking of very large data files IS31 Semantic checking of very large data files IS32 Basic Migration of RAW to NeXus data IS33 Enhanced migration of RAW to NeXus data

Issues:

Issue 1

Title	IS32 Basic Migration of RAW to NeXus data
Detailed description	obsolescence of RAW files and preference for NeXus as standard
Issue champion	Simon Lambert (STFC)
Other interested parties	Any other parties who are also interested in applying Issue Solutions to their Datasets. Identify the party with a link to their contact page on the SCAPE Sharepoint site, as well as identifying their institution in brackets. Eg: Schlarb Sven (ONB)
Possible Solution approaches	No suitable solutions exist at present. Hence, a suitable RAW-to-NeXus migration mechanism would need to be developed.
Datasets	nexus data files
Solutions	Reference to the appropriate Solution page(s), by hyperlink

Evaluation Objectives	scaleability, automation
Success criteria	Describe the success criteria for solving this issue - what are you able to do? - what does the world look like?
Automatic measures	What automated measures would you like the solution to give to evaluate the solution for this specific issue? which measures are important? If possible specify very specific measures and your goal - e.g. * handle up to 100Gb files without crashing * fail safe - even if it fails, it fails gracefully, rather than crashing, i.e. effective error handling should be in place to allow the environment that hosts the tool to capture the error and notify other services that may interact with the tool. * can migrate very large data files concurrently with migration processing of small files (up to 5 concurrent threads). * can perform migration with a data rate up to 100MB/s
Manual assessment	N.A.
Actual evaluations	links to acutual evaluations of this Issue/Scenario

Issue 2

Title	IS33 Enhanced migration of RAW to NeXus data
Detailed description	desire to enhance the value of the dataset with additional information about an experiment that is not present in the basic data file, so as to enrich the dataset with representation information.
Scalability Challenge	Apart from the file size, volume of content challenges identified in IS29 for nexus files, the raw to nexus format migration tool can be customised to take into account of various other types of experiment data files in the process of the migration. However, the scalability challenge here is that for different instrument (specific to each facility), the other types of experiment data files vary signficantly. This makes it difficult to efficiently migrate large quantity of complex raw data files systematically.
Issue champion	Erica Yang (STFC)
Other interested parties	Any other parties who are also interested in applying Issue Solutions to their Datasets. Identify the party with a link to their contact page on the SCAPE Sharepoint site, as well as identifying their institution in brackets. Eg: Schlarb Sven (ONB)
Possible Solution approaches	push the additional information into metadata fields of a nexus file
Datasets	nexus data files, ICAT catalogue data
Solutions	Reference to the appropriate Solution page(s), by hyperlink

Evaluation Objectives	Not applicable for the 1st and 2nd SCAPE years
Actual evaluations	links to acutual evaluations of this Issue/Scenario

Solutions:

Title	SO 22 Developing a Raw-to-NeXus migration tool
Detailed description	No suitable solutions exist at present. Hence, a suitable RAW-to-NeXus migration mechanism would need to be developed.
Solution Champion	Erica Yang (STFC)
Corresponding Issue(s)	IS32 Basic Migration of RAW to NeXus data IS33 Enhanced migration of RAW to NeXus data
Evaluation	Any notes or links on how the solution performed. This will be developed and formalised by the Testbed SP.

RDST2 Format Migration of (raw) Scientific Datasets

Collection:

Issues:

Issue 1

Issue 2

Solutions:

1 Comment

Miguel Ferreira