Evaluation methodology

This is a description of how to perform an experiment in the SCAPE project, as well as information about the wiki pages and templates that are used.

Evaluation in SCAPE is done through the following steps

1. Define top-10 goals and objectives that will be evaluated
2. For these goal-objective pairs choose what and how to evaluate
   1. Set up an experiment, and execute.
   2. Write an evaluation of findings and results, both measurable as well as non-measurable.

The results will be gathered manually and entered into the evaluation pages on this WIKI. We expect that many of the experiments will be evaluated several times; ultimately until the defined metric goal (1c - see below) is reached. Thereby resulting in multiple findings and results (3a - see below) showing progress of SCAPE developments. See example. For some objectives (e.g. organisational fit), it might not be applicable to define a precise measure, and for these objectives, a more qualitative human understandable evaluation (in form of a report) will be written.

Evaluation guidelines

Background

You might want to have a look at these pages:

• Definition of top-10 goals and objectives
• Metrics catalogue

Testbeds, user stories, experiments and evaluations are found in a tree structure here

How-to

1. Locate/create user story description and verify that information is correct on the wiki page
   • User story template
2. Locate/create experiment description and verify that information is correct on wiki page
   • Experiment template
3. Locate/create platform description and verify that information is correct on wiki page
   • Existing platforms
   • Platform template
4. Locate/create dataset description and verify that information is correct on wiki page
   • Existing datasets
   • Dataset template
5. Locate/create evaluation description and verify that information is correct on wiki page
   • Evaluation template

Evaluation

Each evaluation will use a basic evaluation scheme

1. Set up evaluation page
   1. Define metrics (Metrics Catalogue)
   2. Define metric baseline (ground truth) - e.g. this could be current state for a tool running on a single machine
   3. Define metric goal - what result do we want to achieve
   4. Define non-measurable points
2. Write relevant technical details, use WebDAV where suited (see evaluation template)
3. Write results, and other important information in the evaluation template
   1. Especially results for the defined metrics (1a) must be recorded

Notes (historical information, not part of methodology)

This document finishes the third milestone (MS78: Refined SCAPE evaluation methodology) in work package 18 (TB.WP4: Evaluation of Results) of the SCAPE project - a work package dedicated to evaluating results of experiments carried out during the project.

Methodology changes

We have received feedback on the Draft Evaluation Methodology, using a questionnaire, which gave input to a few changes in the methodology. Also, the replacement of the scenario, solution and issue structure with stories and experiments has impacted the Evaluation Methodology

1. Evaluations is now linked to stories, experiments, dataset and platform.
2. An evaluation can be described by a blog post on the OPF website, just fill out the manual assessment template and paste the link into the document.
3. The platform template has been updated with extra fields to describe parallel execution system like a Hadoop installation and a section describing performance called benchmark.
4. The component and system evaluation template has been has been merged to one template