Skip to end of metadata
Go to start of metadata
IS31 Semantic checking of very large data files
Detailed description In order to ensure that the data to be preserved is of adequate quality , the contents of NeXus data files would need to be validated for their correctness against a given data model. Each data model is specified in a NeXus Definition Language (NXDL) file and contains assertions that define the expected content of a NeXus file. For example, a data model could define a metadata element (key-value pair) called “Integral” to represent the total integral monitor counts for grazing incidence small angle diffractometer GISAS for either x-ray or neutrons. In this scenario, the data type of the metadata element “Integral” would be an integer. For a NeXus data file conforming to this data model, it would be necessary to validate the value(s) assigned to “Integral” to ensure it is of appropriate data type.
Issue champion Simon Lambert (STFC)
Other interested parties
Any other parties who are also interested in applying Issue Solutions to their Datasets. Identify the party with a link to their contact page on the SCAPE Sharepoint site, as well as identifying their institution in brackets. Eg: Schlarb Sven (ONB)
Possible Solution approaches Use of the NeXus validation toolkit - developed and used by the NeXus community - as part of the preservation ingest workflow.
Datasets nexus data files
Solutions Reference to the appropriate Solution page(s), by hyperlink
Evaluation Objectives scaleability, automation
Success criteria Describe the success criteria for solving this issue - what are you able to do? - what does the world look like?
Automatic measures What automated measures would you like the solution to give to evaluate the solution for this specific issue? which measures are important?
If possible specify very specific measures and your goal - e.g.
* handle up to 100Gb files without crashing
* fail safe - even if it fails, it fails gracefully, rather than crashing, i.e. effective error handling should be in place to allow the environment that hosts the tool to capture the error and notify other services that may interact with the tool.
* can check very large data files concurrently with semantic checking of small files (up to 5 concurrent threads).
* can perform semantic checking with a data rate up to 100MB/s
Manual assessment N.A.
Actual evaluations links to acutual evaluations of this Issue/Scenario
issue issue Delete
conformance conformance Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.