Skip to end of metadata
Go to start of metadata
You are viewing an old version of this page. View the current version. Compare with Current  |   View Page History

Metrics Catalogue

To unify metrics across all evaluations all metrics should be registered in this Metrics Catalogue. So - when picking metrics for an evaluation run through the catalogue and pick any already defined or enter a new metric when needed.

NumberOfObjectsPerHour integer Number of objects that can be processed per hour
Could be used both for component evaluations on a single machine and on entire platform setups
IdentificationCorrectnessInPercent integer
Defining a statistical measure for binary evaluations - see detailed specification below 85 %
Between 0 and 100
The max file size a workflow/component has handled
Specify in Gbytes
MinObjectSizeHandledInMbytes integer The min file size a workflow/component has handled - illustrates capability of running on heterogeneous file sizes when combined with MaxObjectSizeHandledInGbytes 20
Specify in Mbytes
PlanEfficiencyInHours integer
Number of hours it takes to build one preservation plan with Plato
Specify in hours
The throughput of data measured in Gybtes per minute
Specify in Gbytes per minute
ThroughputGbytesPerHour integer
The throughput of data measured in Gbytes per hour
Specify in Gbytes per minute
ReliableAndStableAssessment boolean
Manual asessment on if the experiment performed reliable and stable
NumberOfFailedFiles integer
Number of files that failed in the workflow
QAFalseDifferentPercent integer Number of content comparisons resulting in original and migrated different, even though human spot checking says original and migrated similar. 5%
Between 0 and 100
float The average processing time in hours per item
Positive floating point number

An attribute/measure catalogue is also developed in PW - this evaluation metrics catalogue will be merged with the PW catalogue in year-3.

If you want to have a quick glance at the PW catalogue its located here (google docs):

Write to Christhop Becker at [email protected] to ask for access to the google doc

If you already are familiar with the PW catalogue you are off cause most welcome to use already existing metrics from in there - this will make the merging in year-3 much easier. But this is currently NOT a requirement.

Binary evaluation method (FMeasure)

We use sensitivity and specificity as statistical measures of the performance of the binary classification test where 
Sensitivity = Σ true different / (Σ true different + Σ false similar
Specificity = Σ true similar / (Σ true similar + Σ false different
and the F-measure is calculated on this basis as shown in the table below:


This is one suggested way which is nicely applicable if we test for binary correctness of calculations, i.e. it is applicable for characterisation and QA

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.