Skip to end of metadata
Go to start of metadata
You are viewing an old version of this page. View the current version. Compare with Current  |   View Page History


Bolette Jurik (SB)

Evaluation points

In this testbed experiment we focus on performance. The earlier experiment EVAL-LSDR6-1 on mp3 to wav migration and QA using xcorrSound also focused on correctness. Moving the workflow to Hadoop to prove scalability, should not affect correctness of the tool.

  • Scalability The workflow must be able to process a large collection within reasonable time. That is we want to be able to migrate and QA a large collection of radio broadcast mp3-files (20 Tbytes - 175.000 files) within weeks rather than years. The goal of 1000 for NumberOfObjectsPerHour would mean that we can migrate the 20TB radio broadcast mp3 collection in a week.
  • Reliability The workflow must run reliably without failing on a large number of files, and it must be possible to restart the workflow without loosing work.
  • Correctness/Scalability We must believe to some extent that the automatic QA correctly identifies the "questionable" migrations, such that these can be checked in a manual QA process. We must however also insist that the number of migrations to check manually is minimal, as this is a very resource demanding process. The goal for QAFalseDifferentPercent has been changed to 2%. This means that we would have to check 3500 migrated 2 hour wav files manually. This is already too resource demanding. However the poor quality of the original files is a great challenge for the content comparison tool, and it turns out this is also too much to ask!

This table shows the results of the evaluation of the mp3 to wav migration and QA performed on the SB Hadoop Cluster.

Assessment of measurable points

Metric Description Metric baseline Metric goal Evaluation 2014 April 8th* Evaluation 2014 June 17th-23rd**
NumberOfObjectsPerHour Performance efficiency - Capacity / Time behaviour 18 (9th-13th November 2012) 1000 204
QAFalseDifferentPercent Functional suitability - Correctness 0.412 % (5th-9th November 2012) 2%

*Based on the small experiment from April with max split size 128 below.

**Based on the large scale experiment from June below.

Discussion and Conclusion

The conclusion is that the workflow does scale! We will not be able to migrate the collection in 1 week, but we will be able to do it in one month on the SB Hadoop cluster, which is considerably better than the one year needed without Hadoop (last evaluation).

The conclusion on correctness is really more of a discussion... The measure QAFalseDifferentPercent is defined as "Ratio of 'QA decided different'/'human judged same', that is ratio of content comparisons resulting in original and migrated different, even though human evaluation define original and migrated similar". As the large scale evaluations were performed on .5TB input, comprising 4998 2-hour mp3 files or 416.5 days, that is over a year of audio, we did not annotate the input, which means that ~8.7% is the number of files, where our audio content comparison tool reported that content of the original and the migrated files were not similar. Some of these may be actual migration errors. Most of them we however believe is due to the poor quality of the original material. Some of the original mp3 files have long periods of "nothing recorded" or silence. Silence or "almost silence" is very difficult to compare, and the tool will report not similar on these files. A better output would probably be "too much silence to perform content comparison". I would like to refer to the correctness evaluation of the xcorrSound waveform-compare tool done last year instead. TODO REF

Small Experiments April 2014

All run on a file list of 58 files (7.2Gb in total).

max-split-size duration
launched maps
37m, 58.593s = 2278.593s
24m, 1.9s = 1441.9s
18m, 17.917 = 1097.917
17m, 3.176 = 1023.176
16m, 54.703s = 1014.703s
17m, 29.96s = 1049.96

The small experiments were mainly run to decide an optimal max-split-size (or an optimal number of map-reduce map tasks). The exact number of MR map tasks seem not to have a big influence on performance, as long as we have more than 12. That is as long as max split size is at most 256 on an input file list of 58 files. We note that we get approximately twice as many launched maps for the waveform-compare Hadoop job, simply because the input list is approximately twice as big, as it is a list of pairs. We can of course adjust this to get approximately the same number of jobs, but it does not seem to be important for the performance.

The large scale experiments were held up for a while, due to too few connection to storage. As this job writes very much data, the Isilon disk I/O and CPU use were being maxed out, even though we were trying to "play nice" and only run 24 maps concurrently. The number of connections to the 16 nodes Isilion storage solution at SB were 2 when the small scale experiments were run. It was then set up to five connections before we ran the large scale experiments.

Large Scale Experiments June 2014

This line of tests will focus on scalability. If max-split-size=256 (bytes) gives us 2*12 maps on an input-txt-file with 58 files, this means 256 bytes is approx 58/12=4,8333 files, so one file is approx 256/4.8333=52.9655 bytes. Then if we want approx 2*12 maps on an input-txt-file with 1000 files, we want max-split-size to be approx 1000/12*52.9655 = 4413.7931 bytes.

The jobs were run on file-lists of approximately 1000 files (100MB); the max.split.size was set to 4414; and each job writes approximately 3.1TB of intermediate and output wav files (+ some small log files).

date size:#mp3s total size duration total duration NumberOfObjectsPerHour failure total failure QAFalseDifferentPercent
2014 Jun 17 1000 1000 4h, 33m 4h, 33m 220
63 63 6.3
2014 Jun 18 1000 2000 4h, 23m 8h, 56m 224
111 174 8.7
2014 Jun 19 999 2999 4h, 20m 13h, 29m 222
52 226 ~7.5
2014 Jun 20 1000 3999 4h, 27m 17h, 56m 223
142 368 ~9.2
2014 Jun 23 999 4998 4h, 28m 22h, 24m 223
67 435 ~8.7

Assessment of non-measurable points

In the last evaluation, we did include ReliableAndStableAssessment Reliability - Runtime stability in the evaluation points, and we wrote true both in baseline value (Manual assessment: the experiment performed reliably and stably for 13 days, but then Taverna failed with  java.lang.OutOfMemoryError: Java heap spacedue to /tmp/ being filled up. All results were however saved, and the workflow could simply be restarted with a new starting point in the input list) and in goal. This time I could simply write true again, as this experiment also performed reliable and stably for around 4 hours. I will however note that this experiment was not focused on reliability, and all intermediate results are potentially lost if the workflow is killed. I will also note that we partitioned the input to the workflow, so it worked on only 1000 files at a time. This was done as the test environment had on upper limit on available storage, and the workflow produces approximately 3.1TB of output files for each 1000 input files. The workflow will fail if it does not have enough output storage.

A note about goals-objectives omitted, and why

This evaluation covers performance, reliability and functional suitability to some extent. We did not look at the metrics MaxObjectSizeHandledInGbytes and MinObjectSizeHandledInMbytes. These measures would certainly contribute to the evaluation. Our collection (Danish Radio broadcasts, mp3) has mp3 files varying very little in size (approx. 2 hours, average file size 118Mb, largest file: 124Mb) and the workflow thus produces wav files varying very little in size (2 wav files of around 1.4Gb for one 118Mb mp3 file). The test mp3 files used under development were of course considerably smaller (around 7Mb) and produced smaller output (around 50Mb*2 per mp3 file). We think that the workflow can handle larger files as well, but this was not tested. We can report that for input MinObjectSizeHandledInMbytes is around 7Mb and MaxObjectSizeHandledInGbytes is around 0.124Gb. For output MinObjectSizeHandledInMbytes is around 50Mb and MaxObjectSizeHandledInGbytes is around 1.4Gb. This would be an interesting measure to experiment further with.

We did also not look at the metrics ThroughputGbytesPerMinute, ThroughputGbytesPerHour, or AverageRuntimePerItemInHours. These are all possible to compute though. The evaluation 2014 June 17th-23rd gave us NumberOfObjectsPerHour=223. We can turn this around to get AverageRuntimePerItemInHours 1/223 = 0.004484305 hours. The NumberOfObjectsPerHour=223 is nicer for humans to read. To compute the others, we need the throughput size in Gb. Our question here is what throughput means. We wrote that the 1000 files in input were only approximately 100Mb but they produced 3.1Tb of intermediate and output wav files. Half of these (1.55TB) is output, and we will use this as the throughput size. Then for NumberOfObjectsPerHour=223, we 1.55/1000*223 = 0.34565TB or 0.34565×1024 = 353.9456Gb of throughput, that is ThroughputGbytesPerHour~=354Gb, or ThroughputGbytesPerMinute~=354/60=5.9Gb.

The evaluation does not cover

  • Organisational maturity
  • Maintainability
  • Planning and monitoring efficiency
  • Commercial readiness

This experiment was focused on tools and platform and performance, and we will keep the evaluations to the specific experiment.

Technical details

The workflow that was used is version 4 of the Slim Migrate And QA mp3 to Wav Using Hadoop Jobs workflow available from

The Hadoop jobs that were used are from commit e1ec47d of the

The waveform-compare tool that was used was from xcorrSound release v2.0.2

The ffmpeg used was version 0.10 Copyright (c) 2000-2012 the FFmpeg developers built on Mar  9 2012 09:32:12 with gcc 4.4.6 20110731 (Red Hat 4.4.6-3).

The mpg321 used was version 0.2.10. Copyright (C) 2001, 2002 Joe Drew.

The cluster set up that was used was the June 2014 version of the SB Hadoop Platform.


The Taverna logs and outputs of the June experiment are stored on along with the SB scape Hadoop Cluster map-reduce client configuration.

Evaluation notes

Could be such things as identified issues, workarounds, data preparation, if not already included above


Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.