Skip to end of metadata
Go to start of metadata

Evaluator(s)

Tomasz Hoffmann (PSNC)

Evaluation points

The objective of this evaluation task was to test the efficiency of one instance of the DICOM data provider available within the Medical Data Center platform at PSNC. In order to measure download performance the metric named number of objects per second has been used. Other metrics gathered during the tests provide additional information and include throughput in bytes per secondmax object size handled in bytes and min object size handled in bytes. The metric goal has not been determined because there are currently no related requirements defined by the users. This evaluation is composed of 7 tests from which the first 5 are executed using one computer (client) and the last two are executed using two computers (the reason for split into two computers is that one computer can efficiently test MDC platform with maximum 20 concurrent threads).

Assessment of measurable points
Metric Description Metric goal July 1, 2014 [Test 1] July 4, 2014 [Test 2] July 4, 2014 [Test 3]
July 4, 2014 [Test 4]
July 4, 2014 [Test 5]
July 7, 2014 [Test 6]
July 7, 2014 [Test 7]
number of objects per second number of downloaded DICOM files* per second - 6.08 [obj/s] 10.01 [obj/s]
13.82 [obj/s]
14.74 [obj/s]
15.59 [obj/s]
9.50 + 14.36 [obj/s] 11.53 + 16.06 [obj/s]
throughput in bytes per second
_bytes per second_ - 2 854 804 [bytes/s]
4 644 810 [bytes/s]
6 413 019 [bytes/s]
6 839 388 [bytes/s]
7 233 589 [bytes/s]
4 410 289 + 6 684 607 [bytes/s] 5 350 649 + 7 489 928 [bytes/s]
max object size handled in bytes
max size of DICOM file - 1 827 084 [bytes]
1 827 084 [bytes]
1 827 084 [bytes]
1 827 084 [bytes]
1 827 084 [bytes]
1 827 084 [bytes]
1 827 084 [bytes]
min object size handled in bytes
min size of DICOM file - 44 090 [bytes]
44 090 [bytes]
44 090 [bytes]
44 090 [bytes]
44 090 [bytes]
44 090 [bytes] 44 090 [bytes]

Note: *an object is a single DICOM file  

Metrics must be registered in the metrics catalogue 

Visualisation of results

The chart below shows relation between download speed (in objects per second) and the number of download threads used in the test. First 5 tests show that a single client can reach up to approx. 16 objects per second, which is in fact the limit of the client computer, as when using two client computers it was possible to retrieve around 27 objects per second (with 40 threads).

  

Tables below provide additional statistics. Table 1. presents a summary of all files and series used in the test (please note that a single series is composed of multiple DICOM files related to a single patient's examination, e.g. CT scan, RTG). Table 2. and Table 3 presents overall statistics related to series. Table 4. and Table 5. presents overall statistics related to files (objects).

Table 1. Overall statistics of the tests

Parameter Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7
Number of threads 1 5 10 15 20 10 + 10 20 + 20
Number of series 
263 263 263 263 263 262 + 262 262 + 262
Number of DICOM files 
27 053 27 053 27 053 27 053 27 053 27 051 + 25 881 27 051 + 25 761
Total size of DICOM files
12 550 [MB]
12 550 [MB]
12 550 [MB]
12 550 [MB]
12 550 [MB]
12 547 + 12 105 [MB] 12 547 + 12 013 [MB]
Total download time
4 448 [s]
2 702 [s]
1 957 [s]
1 835 [s]
1 735 [s]
2 845 + 1 802 [s] 2 345 + 1 604 [s]
Average serie download speed (series per second) 0.05 [series/s] 0.09 [series/s]
0.13 [series/s]
0.14 [series/s]
0.15 [series/s]
0.09 + 0.14 [series/s] 0.11 + 0.16 [series/s]

Table 2. Overall statistics related to series (maximum values)

Parameter Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7
Maximum series size
233 [MB] 233 [MB] 233 [MB]
233 [MB]
233 [MB]
233 + 233 [MB] 233 + 233 [MB]
Maximum series download time
73 [s] 165 [s]
223 [s] 303 [s] 373 [s] 291 + 233 [s] 463 + 383 [s]

Table 3. Overall statistics related to series (minimum values)

Parameter Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7
Minimum series size 
0.04 [MB]
0,04 [MB] 0,04 [MB]
0,04 [MB]
0,04 [MB]
0.04 + 0.04 [MB] 0.04 + 0.04 [MB]
Minimum series download time 
0.09 [s] 0.11 [s] 0.12 [s] 0.13 [s] 0.19 [s] 0.17 + 0.10 [s] 0.22 + 0.14 [s]

Table 4. Overall statistics related to files (maximum values)

Parameter Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7
Maximum file size 
1.82 [MB] 1.82 [MB] 1.82 [MB]
1.82 [MB]
1.82 [MB]
1.82 + 1.82 [MB]
1.82 + 1.82 [MB]
Maximum file download time 
5.24 [s]
5.63 [s] 5.53 [s] 5.59 [s] 7.36 [s] 2.21 + 5.27 [s] 4.32 + 5.49 [s]

Table 5. Overall statistics related to files (minimum values)

Parameter Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7
Minimum file size  
0.04 [MB]   
0.04 [MB] 0.04 [MB]
0.04 [MB]
0.04 [MB]
0.04 + 0.04 [MB] 0.04 + 0.04 [MB]
Minimum file download time  
0.09 [s]
0.09 [s] 0.10 [s] 0.10 [s] 0.11 [s] 0.12 + 0.10 [s] 0.16 + 0.10 [s]

Raw log files

All log files created during the evaluation are available online at:

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/dicom-test-results/access/test01_01_07_2014.log

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/dicom-test-results/access/test02_04_07_2014.log

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/dicom-test-results/access/test03_04_07_2014.log

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/dicom-test-results/access/test04_04_07_2014.log

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/dicom-test-results/access/test05_04_07_2014.log

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/dicom-test-results/access/test06_07_07_2014.log

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/dicom-test-results/access/test06_07_07_2014.log

Technical details

Workflow

The experiment was composed of the following steps:

  1. prepare data set of DICOM files (over ~10GB, which is amount of data produced by WCPT hospital in one day),
  2. set the number of downloading threads [dicomClient.py]
  3. download data from MDC server, and log on information about: [dicomClient.py]
    1. size of recived DICOM file
    2. time of reciving data for each DICOM file
  4. parse a log file in order to evaluate metrics (for metrics description please see previous point) [dicomClientResultParser.py]

Scripts used to execute evaluation

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/python/DICOM-tests/dicomClient.py

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/python/DICOM-tests/dicomClientResultParser.py

Execution commands

 python dicomClient.py
 python dicomClientResultParser.py dicomClientStats.txt dicomClientTime.txt test08_results.txt

Note: Scripts are prepared for Python in version 2.7.

Labels:
None
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.