Evaluator(s)
Tomasz Hoffmann (PSNC)
Evaluation points
The objective of this evaluation task was to test the efficiency of one instance of the DICOM data provider available within the Medical Data Center platform at PSNC. In order to measure download performance the metric named number of objects per second
has been used. Other metrics gathered during the tests provide additional information and include throughput in bytes per second, max object size handled in bytes and min object size handled in bytes. The metric goal has not been determined because there are currently no related requirements defined by the users. This evaluation is composed of 7 tests from which the first 5 are executed using one computer (client) and the last two are executed using two computers (the reason for split into two computers is that one computer can efficiently test MDC platform with maximum 20 concurrent threads).
Assessment of measurable points
| Metric | Description | Metric goal | July 1, 2014 [Test 1] | July 4, 2014 [Test 2] | July 4, 2014 [Test 3] |
July 4, 2014 [Test 4] |
July 4, 2014 [Test 5] |
July 7, 2014 [Test 6] |
July 7, 2014 [Test 7] |
|---|---|---|---|---|---|---|---|---|---|
| number of objects per second |
number of downloaded DICOM files* per second | - | 6.08 [obj/s] | 10.01 [obj/s] |
13.82 [obj/s] |
14.74 [obj/s] |
15.59 [obj/s] |
9.50 + 14.36 [obj/s] | 11.53 + 16.06 [obj/s] |
| throughput in bytes per second |
_bytes per second_ | - | 2 854 804 [bytes/s] |
4 644 810 [bytes/s] |
6 413 019 [bytes/s] |
6 839 388 [bytes/s] |
7 233 589 [bytes/s] |
4 410 289 + 6 684 607 [bytes/s] | 5 350 649 + 7 489 928 [bytes/s] |
| max object size handled in bytes |
max size of DICOM file | - | 1 827 084 [bytes] |
1 827 084 [bytes] |
1 827 084 [bytes] |
1 827 084 [bytes] |
1 827 084 [bytes] |
1 827 084 [bytes] |
1 827 084 [bytes] |
| min object size handled in bytes |
min size of DICOM file | - | 44 090 [bytes] |
44 090 [bytes] |
44 090 [bytes] |
44 090 [bytes] |
44 090 [bytes] |
44 090 [bytes] | 44 090 [bytes] |
Note: *an object is a single DICOM file
Metrics must be registered in the metrics catalogue
Visualisation of results
The chart below shows relation between download speed (in objects per second) and the number of download threads used in the test. First 5 tests show that a single client can reach up to approx. 16 objects per second, which is in fact the limit of the client computer, as when using two client computers it was possible to retrieve around 27 objects per second (with 40 threads).
Tables below provide additional statistics. Table 1. presents a summary of all files and series used in the test (please note that a single series is composed of multiple DICOM files related to a single patient's examination, e.g. CT scan, RTG). Table 2. and Table 3 presents overall statistics related to series. Table 4. and Table 5. presents overall statistics related to files (objects).
Table 1. Overall statistics of the tests
| Parameter | Test 1 | Test 2 | Test 3 | Test 4 | Test 5 | Test 6 | Test 7 |
|---|---|---|---|---|---|---|---|
| Number of threads | 1 | 5 | 10 | 15 | 20 | 10 + 10 | 20 + 20 |
| Number of series |
263 | 263 | 263 | 263 | 263 | 262 + 262 | 262 + 262 |
| Number of DICOM files |
27 053 | 27 053 | 27 053 | 27 053 | 27 053 | 27 051 + 25 881 | 27 051 + 25 761 |
| Total size of DICOM files |
12 550 [MB] |
12 550 [MB] |
12 550 [MB] |
12 550 [MB] |
12 550 [MB] |
12 547 + 12 105 [MB] | 12 547 + 12 013 [MB] |
| Total download time |
4 448 [s] |
2 702 [s] |
1 957 [s] |
1 835 [s] |
1 735 [s] |
2 845 + 1 802 [s] | 2 345 + 1 604 [s] |
| Average serie download speed (series per second) | 0.05 [series/s] | 0.09 [series/s] |
0.13 [series/s] |
0.14 [series/s] |
0.15 [series/s] |
0.09 + 0.14 [series/s] | 0.11 + 0.16 [series/s] |
Table 2. Overall statistics related to series (maximum values)
| Parameter | Test 1 | Test 2 | Test 3 | Test 4 | Test 5 | Test 6 | Test 7 |
|---|---|---|---|---|---|---|---|
| Maximum series size |
233 [MB] | 233 [MB] | 233 [MB] |
233 [MB] |
233 [MB] |
233 + 233 [MB] | 233 + 233 [MB] |
| Maximum series download time |
73 [s] | 165 [s] |
223 [s] | 303 [s] | 373 [s] | 291 + 233 [s] | 463 + 383 [s] |
Table 3. Overall statistics related to series (minimum values)
| Parameter | Test 1 | Test 2 | Test 3 | Test 4 | Test 5 | Test 6 | Test 7 |
|---|---|---|---|---|---|---|---|
| Minimum series size |
0.04 [MB] |
0,04 [MB] | 0,04 [MB] |
0,04 [MB] |
0,04 [MB] |
0.04 + 0.04 [MB] | 0.04 + 0.04 [MB] |
| Minimum series download time |
0.09 [s] | 0.11 [s] | 0.12 [s] | 0.13 [s] | 0.19 [s] | 0.17 + 0.10 [s] | 0.22 + 0.14 [s] |
Table 4. Overall statistics related to files (maximum values)
| Parameter | Test 1 | Test 2 | Test 3 | Test 4 | Test 5 | Test 6 | Test 7 |
|---|---|---|---|---|---|---|---|
| Maximum file size |
1.82 [MB] | 1.82 [MB] | 1.82 [MB] |
1.82 [MB] |
1.82 [MB] |
1.82 + 1.82 [MB] |
1.82 + 1.82 [MB] |
| Maximum file download time |
5.24 [s] |
5.63 [s] | 5.53 [s] | 5.59 [s] | 7.36 [s] | 2.21 + 5.27 [s] | 4.32 + 5.49 [s] |
Table 5. Overall statistics related to files (minimum values)
| Parameter | Test 1 | Test 2 | Test 3 | Test 4 | Test 5 | Test 6 | Test 7 |
|---|---|---|---|---|---|---|---|
| Minimum file size |
0.04 [MB] |
0.04 [MB] | 0.04 [MB] |
0.04 [MB] |
0.04 [MB] |
0.04 + 0.04 [MB] | 0.04 + 0.04 [MB] |
| Minimum file download time |
0.09 [s] |
0.09 [s] | 0.10 [s] | 0.10 [s] | 0.11 [s] | 0.12 + 0.10 [s] | 0.16 + 0.10 [s] |
Raw log files
All log files created during the evaluation are available online at:
Technical details
Workflow
The experiment was composed of the following steps:
- prepare data set of DICOM files (over ~10GB, which is amount of data produced by WCPT hospital in one day),
- set the number of downloading threads [dicomClient.py]
- download data from MDC server, and log on information about: [dicomClient.py]
- size of recived DICOM file
- time of reciving data for each DICOM file
- parse a log file in order to evaluate metrics (for metrics description please see previous point) [dicomClientResultParser.py]
Scripts used to execute evaluation
Execution commands
python dicomClient.py python dicomClientResultParser.py dicomClientStats.txt dicomClientTime.txt test08_results.txt
Note: Scripts are prepared for Python in version 2.7.