Skip to end of metadata
Go to start of metadata

Evaluator(s)

Tomasz Hoffmann (PSNC)

Evaluation points

The objective of this evaluation task was to test the efficiency of one instance of the DICOM data receiver available within the Medical Data Center platform at PSNC. In order to measure ingest performance the metric named number of objects per second has been used. Other metrics gathered during the tests provide additional information and include throughput in bytes per secondmax object size handled in bytes and min object size handled in bytes. The metric goal was set to 0.25 objects per second to make sure that it is possible to ingest 10GB of data per day, which is the approx. amount of data produced by WCPT hospital each day. This evaluation is composed of 3 tests for 1, 4 and 10 concurrent threads sending DICOM files from one computer (client). This evaluation included copying data to archiving system.

Assessment of measurable points
Metric Description Metric goal June 6, 2014 [1T] June 25, 2014 [4T]
June 25, 2014 [10T]
number of objects per second number of ingested DICOM files* per second 0.25 [obj/s] 1 x 1.17 = 1.17 [obj/s] 4 x 0.31 = 1.24 [obj/s] 10 x 0.12 = 1.2 [obj/s]
throughput in bytes per second
_bytes per second_ 116 000 [bytes/s] 514 483 [bytes/s]
145 763 [bytes/s] 54 645 [bytes/s]
max object size handled in bytes
max size of DICOM file - 1 502 102 [bytes]
1 827 084 [bytes] 1 502 102 [bytes]
min object size handled in bytes
min size of DICOM file - 201 554 [bytes]
44 090 [bytes] 44 406 [bytes]

Notes:

  • an object is a single DICOM file
  • xT - means x sending threads
  • results in table presents statistics per one sending thread that are multiplied by the number of threads used in the test to get overall statistic

Metrics must be registered in the metrics catalogue

Visualisation of results

The chart below shows relation between ingest speed (in objects per second) and the number of sending threads used in the test. Tests show that a single client can reach up to approx. 1,2 objects per second, which is in fact limited not by the client computer but by the server (quite long time of storing files in the archiving system).

 

Tables below provide additional statistics. Table 1. presents a summary of all files and series used in the test (please note that a single series is composed of multiple DICOM files related to a single patient's examination, e.g. CT scan, RTG). Table 2. and Table 3 presents overall statistics related to series. Table 4. and Table 5. presents overall statistics related to files (objects).

Table 1. All files and series stats

Parameter Test 1 Test 2 Test 3
No sending threads 1 4 10
All series number
33 262
66
All DICOM files number
3793 27076
6479
All DICOMs size
1658 [MB]
12564 [MB]
2901 [MB]
Saving time
3223 [s]
86196 [s]
53084 [s]
HDFS saving time
382 [s]
3285 [s]
1182 [s]
Hbase saving time
49 [s] 28 [s]
410 [s]
Sftp saving time
2722 [s]
81973 [s]
51402 [s]

Table 2. Maximum series stats

Parameter Test 1 Test 2 Test 3
No sending thread
1 4 10
Max series size
309 [MB] 233 [MB] 211 [MB]
Max series sending time
704 [s] 1495 [s] 3816 [s]
Max series hdfs saving time
60 [s] 70 [s] 84 [s]
Max series hbase saving time
10 [s] 18 [s] 36 [s]
Max series sftp saving time
620 [s] 1424 [s]
3694 [s]
Max series items number
587 444 401

Table 3. Minimum series stats

Parameter Test 1 Test 2 Test 3
No sending thread 
1 4 10
Min series size 
0.24 [MB]
0.04 [MB]
0.04 [MB] 
Min series sending time 
0.39 [s] 2.27 [s] 3.80 [s]
Min series hdfs saving time 
0.08 [s] 0.06 [s] 0.06 [s]
Min series hbase saving time 
0.01 [s] 0.006 [s] 0.01 [s]
Min series sftp saving time 
0.24 [s] 2.13 [s]
3.50 [s]
Min series items number 
1 1

Table 4. Maximum files stats

Parameter Test 1 Test 2 Test 3
No sending thread 
1 4 10
Max file size 
1.50 [MB] 1.82 [MB]
1.50 [MB] 
Max file sending time 
24 [s]
14[s]
36 [s]
Max file hdfs saving time 
1 [s]
5 [s]
2 [s]
Max file hbase saving time 
1 [s]
1 [s]
1 [s] 
Max file sftp saving time 
23 [s]
13 [s]  
35 [s]

Table 5. Minimum files stats

Parameter Test 1 Test 2 Test 3
No sending thread  
1 4 10
Min file size  
0.20 [MB]   
0.04 [MB] 
0.04 [MB]  
Min file sending time  
0.16 [s]
0.98 [s] 1.10 [s] 
Min file hdfs saving time  
0.06 [s] 0.04 [s] 0.05 [s] 
Min file hbase saving time  
0.01 [s] 0.01 [s] 0.01 [s] 
Min file sftp saving time  
0.06 [s] 0.85 [s] 
0.90 [s] 

Raw log files

All log files created during the evaluation are available online at:

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/dicom-test-results/ingest/with_sftp/test01_6_06_2014.log

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/dicom-test-results/ingest/with_sftp/test02_25_06_2014.log

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/dicom-test-results/ingest/with_sftp/test03_25_06_2014.log

Technical details

Workflow

The experiment is composed of the following steps:

  1. prepare data set of DICOM files (over ~10GB, which is amount of data produced by WCPT hospital in one day),
  2. send data to MDC server ( [dicomSender.py]), and log on the server information about:
    1. size of recived DICOM file
    2. time of saving data in:
      1. HDFS
      2. Hbase
      3. SFTP
  3. parse server log file in order to evaluate metrics [resultsParser.py] (for metrics description please see previous point)

Scripts used to execute evaluation

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/python/DICOM-tests/dicomSender.py

https://git.man.poznan.pl/stash/projects/SCAP/repos/test-scripts/browse/python/DICOM-tests/resultsParser.py

Execution commands

python dicomSender.py ~/tmp/dicom/

python resultsParser.py ~/tests/dicom_tests/1_07_2014/dcmreceiverStats.log test01_1_07_2014.txt

Notes:

  • scripts are prepared for Python in version 2.7.
  • dcmreceiverStats.log is a dcmrcv tool log file 
Labels:
None
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.