IS27 Quality assurance in redownload workflows of digitised books

| *Title* \\ | Quality assurance in redownload workflows of digitised books \\ |
| *Detailed description* | Cultural heritage institutions such as libraries, museums and archives have been carrying out large scale digitisation projects during the last decade. The production of digitised versions of books or newspapers took either place inhouse or it was outsourced to an external partner. However, even if commercial partners were involved in the production of digital masters, usually, the results and any attached property rights had mostly been transferred entirely to the originator's institution. \\
\\
These circumstances have changed in some public-private partnerships, where the digitisation is carried out by the commercial partner which keeps the original master copy and then produces derivates which are provided to the cultural heritage institution. As a consequence, from the point of view of the cultural heritage institiution, the preservation challenges relate to the derivates rather than the original master copies (considering the very unlikely event that the commercial company disappears together with the digital master copies). \\
\\
This changes an important parameter regarding the use of long term preservation repositories. Instead of producing a master copy once which is stored "forever" in the repository and not supposed to change in future, new derivates of master copies are continuously being made available. The derivates can be downloaded and ingested into the repository as a new version which is either added or replaces the original derivate. \\
\\
In the concrete context of this issue, there are mainly three objects available: \\
# A METS container for each book item.
# A series of digital images (JPEG2000) for each page of the book.
# An hOCR file containing text and layout information from the OCR. \\
All the three object types provide information that can be used in a quality assurance process that helps to determine if a new derivate is better in terms of quality compared to previous versions. First, images can be used for image analysis and comparison, context information from the METS file can be taken to compare images from one book (possibly duplicated pages) or from different versions against each other. Second, hOCR files can be used for doing text content and layout analysis. Finally, a hybrid approach, using image comparision and text/layout analysis can be used. \\ |
| *Scalability Challenge* \\ | Currently there are about \\
* 50.000 books (at least 320 pages each)
* 16 Mio pages (one image and one hOCR file each). \\
For example, a simple compression process that takes 2 seconds for each hOCR file would last 185 days on one single processing node. \\ |
| *[Issue champion|SP:Responsibilities of the roles described on these pages]* | [Schlarb Sven|https://portal.ait.ac.at/sites/Scape/_layouts/userdisp.aspx?ID=32] (ONB). \\ |
| *Other interested parties* \\ | |
| *Possible Solution approaches* | * Text mining for detecting significant changes between stored derivate and new derivate.
* Comparing old and new instances of the corresponding items (books and/or book pages). |
| *Context* | ONB's ABO project (Private public partnership with Google - Google books) \\ |
| *Lessons Learned* | \\ |
| *Training Needs* | \\ |
| *Datasets* | ONB's Google books test data set (Restricted access), 50 selected books with around 30.000 pages (~600 pages per book) sample. \\ |
| *Solutions* | \\ |

h1. Evaluation

| *Objectives* | The objectives are to measure
* _scalability_ in terms of throughput (books/time) related to defined quality assurance workflows with increasing sample size (50, 500, 5000 books) in various steps up to a very large data set (50000 books).
* _reliability_ in terms of error free processing of defined quality assurance workflows.
* _preciseness_ in terms of the number of pages and books correctly identified. |
| *Success criteria* | * Hadoop data preparation (e.g. loading data into HDFS) and quality assurance workflows must be processable in a reasonable amount of time (_scalability_ in terms of throughput). \\
"Reasonable" in this context is a flexible criterium that means "it should rather take days instead of weeks" to process a large book collection like the data set of 50.000 books. To be more concrete, loading all hOCR instances for further analysis into the distributed file system (HDFS) would normally be done once a year depending on the amount of redownloaded items. Given that the number of books will increase over the coming years, loading the data into HDFS should not take longer than one week . The quality assurance workflow should not take longer than around three weeks to run on the complete collection.
* The process should not fail without any reasonable cause, processing errors are explicitely handled and reported by the system (_reliability_).
* The rate of book pairs which are detected as being significantly different is high enough to be used as a tool for manual quality control. "High enough" is measured in terms of sensitivity and specificity thresholds in relation to the evaluation data set like explained below (see Manual assessment). |