Skip to end of metadata
Go to start of metadata

Collection:

Unable to render {include} Couldn't find a page to include called: mp3 (128kbit) with Danish Radio broadcasts

Issue:

Title
IS21 Migration of mp3 to wav
Detailed description A large collection of mp3-files (20 Tbytes - 175.000 files) needs to be migrated to wav. Because of the large amount of data we need automatic QA algorithms to ensure that migration went well.
Scalability Challenge
Large amounts of data (20 Tbytes - 175.000 files) - requires both much I/O and CPU for the migration part as well as I/O and CPU for the QA part. Must be carried out on a distributed platform to be realistic.
Issue champion Bjarne Andersen (SB)
Other interested parties
 
Possible Solution approaches
  • SB: develop an audio comparison tool that based on fast-fourier-transformations (FFT) can compare two waveforms
Context
Lessons Learned  
Training Needs
Datasets mp3 (128kbit) with Danish Radio broadcasts
Solutions SO2 xcorrSound QA audio comparison tool
SO4 Audio mp3 to wav Migration and QA Workflow

Evaluation

Objectives This is about scaleability and robustness. 175.000 files need to be migrated and automatically checked content-wise and format-wise
Success criteria We will have a workflow that converts from mp3 to wav. Checks the migrated files for wav-conformance and checks the actual content by comparing the content of the mp3s and the corresponding wavs to ensure that nothing went wrong
Automatic measures 1. Process 20 files per hour per node
Manual assessment 1. If any of the migrated files does not pass the QA-test they should actually have failed the migration.
Actual evaluations links to acutual evaluations of this Issue/Scenario

Solutions:

Title SO4 Audio mp3 to wav Migration and QA Workflow
Detailed description Informally the workflow is as follows:

SB currently owns a small collection of real audio filer files (digitised cd’s). They are part of the Danish publications that SB preserves. The rest of the Danish cd collection is in WAV. This format has been chosen as the preservation format as this is a raw format, which needs less interpretation or fewer layers of interpretation to be understood by humans and it is also a robust format.

The Danish Radio Broadcast mp3 files from the mp3 (128kbit) with Danish Radio broadcasts dataset are also to be migrated to WAV according to policy. The actual migration will be done using FFmpeg which is one of the SCAPE Action Services recommended tools. The QA will be split into a number of steps. The first step is validation that the migrated file is a correct file in the wanted format. This is done using JHOVE2 to analyse and provide a JHOVE2 property xml file, and next using a Beanshell in Taverna to check that the jhove2 feature “isValid” is true. The second step compares the header information properties of the original and the migrated files to see if they are ‘close enough’. This is done using FFprobe to extract header information and Taverna Beanshells to compare the extracted properties. Another step could be to extract more properties by ‘playing’ the two files.

The third step uses an analysis tool comparing the sound waves. To do this we have to ‘play’ or interpret the mp3 files. Just as a human needs to ‘play’ or interpret the files to hear the sound. A human cannot look at fileA and tell if it is correct or corrupted. We choose a player P and define ‘fileA played on player P’ to be correct. A small randomly chosen subset of files will be played on player P and checked by human ears to be correct making this definition probable. The player used in this workflow is MPG321. Note that MPG321 is an independent implementation of an mp3-decoder – thus independent from FFmpeg, which is used to actually migrate the file. The result of playing fileA on player P (when noone is listening) is a WAV file. The migrated file is already a WAV file, and we can compare the two files using the analysis tool xcorrSound/migrationQA, see SO2 xcorrSound QA audio comparison tool..

All the used tools are wrapped as SCAPE Web Services and Taverna Workflows.

In short:
  • mp3 files are migrated to wav using FFmpeg
  • QA
    1. validation that the migrated file is a correct file in the wanted format using JHOVE2
    2. extraction and comparison of level 1 (header) properties of original and migrated file using FFprobe and Beanshell
    3. play original file using MPG321 and compare the waveforms and output similarity-factor using xcorrSound/migrationQA QA sound comparison tool; see SO2 xcorrSound QA audio comparison tool
Solution Champion
Bolette Jurik (SB)
Corresponding Issue(s)
myExperiment Links
The full workflow is MP3 to WAV Migration +QA. The workflow is combined of a number of smaller workflows, that can also be used or combined in other workflows and solutions:
Tool Registry Links
The workflow uses the following tools
Evaluation
This Migration and QA solution has been developed as a Taverna workflow using web services. This puts focus on availability rather than scalability. The sparse tests run i February 2012 have also been run through Taverna. TODO IN DIRE NEED OF AN UPDATE!!!

We tested the Mp3 to Wav Migrate Validate Compare Workflow on file P1_1000_1200_890106_001.mp3 from the mp3 (128kbit) with Danish Radio broadcasts testbed dataset. The file is 112 Mbyte and the duration is 2 hours,
2 minutes and 5.23 seconds. The workflow was run from Taverna on a local work station using the web services deployed on the SB iapetus test machine. The total time for the workflow is 2.3 minutes, and the most expensive nested workflow is the
JHove2Validate workflow with 1.3 minutes, closely followed by the FFmpegMigrate workflow with 59.2 s. The FFprobeExtractCompare workflow with 12.9 seconds seems to be the cheaper QA in this set-up. The result of the workflow is a migrated
WAV file, and a report that it is valid and that the extracted properties have been preserved.

Running the workflow on 4 additional files from the dataset gave similar results of between 2.2 and 2.3 minutes, valid migrated WAV files and properties preserved. The 6th test run also gave a result wav file, but we could not hear the file, and neither
JHOVE2 or FFprobe was able to read the file. Interestingly the nested FFprobe workflow failed, while the nested JHove2 workflow finished, but without output. At the intermediate values, we see the JHOVE2 SCAPE web service is unable to read the
migrated file. This raises a question of error handling in the Taverna workflows. The nice feature is that in this layered approach, it is still easy to pinpoint where the error originated, but we may want to look at more consistent error handling. Maybe the
workflow should not fail in this case, but rather give meaningful output about a failed web service or beanshell.

We note that we are working on 2-hour sound files. The average file size of the original mp3 files is only 118Mb, but the migrated wav files are approximately 1.4Gb. This means we can probably not hope to improve much on the performance of the
actual FFmpeg migration of the individual files. The mp3 (128kbit) with Danish Radio broadcasts collection is 20 TB and around 150000 files. This means that running the basic workflow migrations sequentially on the test machine would take around 300
days. We can however hope to improve by using the Scape execution platform instead of doing the migrations sequentially.

The migrationQA workflow was tested separately, see SO2 xcorrSound QA audio comparison tool.

The full MP3 to WAV Migration + QA workflow has been tested on a shorter file using the ONB webservices. The input mp3 file was the Creative Commons example file http://ia700308.us.archive.org/21/items/Radiators2008-09-19/Rads2008-19-09-d1t02_vbr.mp3. The total time was 47.9 seconds. The time for FFmpegMigrate 12.2 seconds. For FFprobeExtractCompare 5.5 seconds. For JHove2Validate 9.6 seconds. And for migrationQA comparison including MPG321 decoding 31 seconds.The output wav file is available from http://fue.onb.ac.at/scape/tmp/Rads2008-19-09-d1t02_vbr.mp3_SCAPEFFMPEG25305Service_wavFile_4413676179244047691.wav. The JHOVE2 validation, the FFprobe comparison and the migrationQA comparison returned true, and the output of the migrationQA comparison looks like this:


block 1: 0.999998 1152
block 2: 1 1152
block 3: 1 1152
block 4: 1 1152
block 5: 1 1152
block 6: 1 1152
block 7: 1 1152
block 8: 1 1152
block 9: 1 1152
block 10: 1 1152
block 11: 1 1152
block 12: 1 1152
block 13: 1 1152
block 14: 1 1152
block 15: 1 1152
block 16: 1 1152
block 17: 1 1152
block 18: 1 1152
block 19: 1 1152
block 20: 1 1152
block 21: 1 1152
block 22: 1 1152
block 23: 1 1152
block 24: 1 1152
block 25: 1 1152
block 26: 1 1152
block 27: 1 1152
block 28: 1 1152
block 29: 1 1152
block 30: 1 1152
block 31: 1 1152
block 32: 1 1152
block 33: 1 1152
block 34: 1 1152
block 35: 1 1152
block 36: 1 1152
block 37: 1 1152
block 38: 1 1152
block 39: 1 1152
block 40: 1 1152
block 41: 1 1152
block 42: 1 1152
block 43: 1 1152
block 44: 1 1152
block 45: 0.999999 1152
block 46: 0.999978 1152

The solution performs as expected and solves the corresponding issue,but further testing for accuracy and performance is required.

Title SO2 xcorrSound QA audio comparison tool
Detailed description The xcorrSound QA audio comparison tool compares two sound waves, for instance the audio waves of an original audio file and the audio waves of a migrated file. The tool uses the cross correlation function to find the overlap match. This will give us a match score (between 0 and 1) and also an offset in the second file for the match if the audio has been shifted in the migration (we have examples of this happening). This is not a full solution, but a tool used as part of the workflows in full solutions to a number of issues.

Solution Champion
Bolette Jurik (SB)
Corresponding Issue(s)
myExperiment Link
migrationQA SCAPE Web Service Wav File Comparison Workflow
Tool Registry Link
xcorrSound
Evaluation
October-November 2012

Performance efficiency - Capacity / Time behaviour
NumberOfObjectsPerHour: Number of mp3 files migrated and QA'ed.
The QA performed as part of the issue IS21 Migration of mp3 to wav workflow at the time of the baseline test is FFProbe Property Comparison, JHove2 File Format Validation and XCorrSound migrationQA content comparison. The mp3 files are 118Mb on average, and the two wav files produced as part of the workflow are 1.4Gb on average. Thus a baseline value of 10 objects per hour means that we process 1.18Gb per hour and we produce 28Gb per hour (+ some property and log files). The collection that we are targeting is 20 Tbytes or 175.000 files. With the baseline value of 10 we would be able to process this collection in a little over 2 years. The goal value is set at 1000 so we would be able to process the collection in a week.
Evaluation 1 (9th-13th November 2012). Simple parallelisation. Started two parallel workflows using separate jhove2 installations. Both on the same machine. Processed 879+877 = 1756 files in 4 days, 1 hour and 12 minutes. NumberOfObjectsPerHour=18.
Functional suitability - Correctness
QAFalseDifferentPercent: This is a measure of how many content comparisons result in original and migrated different, even though the two files sound the same to the human ear.
The parallel measure QAFalseSimilarPercent is how many content comparisons result in original and migrated similar, even though the two files sound different to the human ear. We have not experienced this - and we do not expect it to happen. We note that this measure is not improved by testbed improvements, but rather by improvements to the XCorrSound migrationQA content comparison tool in the PC.QA work package. The baseline value is 161 in 3190 ~= 5% (test 2nd-16th October 2012). The goal value of 0.1% is set to make manual checking feasible. The collection that we are targeting is 20 Tbytes or 175.000 files. With QAFalseDifferentPercent at 0.5%, we would still need to check 175 2-hour files manually (with the help of the XCorrSound migrationQA tool with the --verbose flag set).
Evaluation 1 (5th-9th November 2012). Processed 728 files in 3 days, 21 hours and 17 minutes = 5597 minutes, which is 5597/728 = 7.7 minutes pr. file in average. The number of files which returned Failure (original and migrated different) is 3 in 728 or 0.412 % of the files. It turns out that these "false negatives" may not be false negatives after all, but actually migration errors that were caught. The actual migration was done with ffmpeg version 0.10, and can be re-created using this version, but it does not happen with ffmpeg versions >= 1.0.

February 2012

The testbed dataset Danish Radio broadcasts, mp3 which is the basis of issue IS21 Migration of mp3 to wav consists of two hour mp3 files (average file size: 118Mb). When these two hour files are migrated to WAV, we get an average file size around 1.4Gb. In February 2012 the migrationQA workflow did not scale nicely to these two hour sound files, but we have run a test on a file cut to 12Mb (about a tenth of the original size) using dd. The Mp3 to Wav Migrate Validate Compare Workflow (see SO4 Audio mp3 to wav Migration and QA Workflow) used only 34 seconds on the cut file. The migration and the file format validation were successful, but the property comparison reported that the files were not 'close enough'. The reason for this is that cutting the file does not change the header information, so the duration of the original cut file is supposedly 2 hours, 2 minutes and 5.23 seconds, while the duration of the migrated file is 13 minutes and 6.38 seconds. Playing the original cut mp3 using the MPG321 Play mp3 to Wav SCAPE Web Service Workflow (see SO4 Audio mp3 to wav Migration and QA Workflow) used 11.7 seconds. The migrationQA SCAPE Web Service Wav File Comparison Workflow took 1.4 minutes. The result was also negative, but an inspection of the output showed that only the last chunk differed, which probably means that FFmpeg and MPG321 handled the cut off differently.

Labels:
lsdr lsdr Delete
scenario scenario Delete
lsdrscenario lsdrscenario Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.