| Title |
IS18 Verify bitstream integrity |
| Detailed description | Problem: bit rot occurs as a result of the (gradual) deterioration of storage media, leading to damage to digital file objects at the bitstream level. Depending on the extent and position of the damaged bits, files affected by bit rot may end up being completely unreadable. Unlike format obsolescence (where a file cannot be read because no readers are available for the used file format), bit rot results in a loss of information that is irreversible. If multiple copies of an object exist, data losses due to bit rot can be recovered by using the intact copies. However, this requires that archived materials are continuously monitored for signs of bit rot (preservation watch), and that affected materials are recovered upon detection (action services). |
| Scalability Challenge |
Solution should be applicable to very large volumes of data |
| Issue champion | Johan van der Knijff (KB) |
| Other interested parties |
STFC created an issue that is very simuilar to this one: IS30 Fixity capturing and checking of very large data files. Maybe these issues should be merged. |
| Possible Solution approaches | The occurrence of bit rot can be monitored by regular checksum checks. Many well-established and stable tools exist for this. The challenge of this issue mainly lies in the effective and efficient deployment of such tools for (very) large digital collections. The scope of this issue may also include guidelines on backup strategies that eliminate the risk of files that are affected by bit rot being unintentionally transferred to backups (thereby overwriting the original, undamaged objects). |
| Context | Details of the institutional context to the Issue. (May be expanded at a later date) |
| Lessons Learned | Notes on Lessons Learned from tackling this Issue that might be useful to inform the development of Future Additional Best Practices, Task 8 (SCAPE TU.WP.1 Dissemination and Promotion of Best Practices) |
| Training Needs | Is there a need for providing training for the Solution(s) associated with this Issue? Notes added here will provide guidance to the SCAPE TU.WP.3 Sustainability WP. |
| Datasets | To be confirmed. |
| Solutions | Reference to the appropriate Solution page(s), by hyperlink |
Evaluation
| Objectives | Reliability, scalability, robustness |
| Success criteria | Describe the success criteria for solving this issue - what are you able to do? - what does the world look like? |
| Automatic measures | Process ## files per second / process ## MB per second |
| Manual assessment | Apart from automated measures that you would like to get do you foresee any necessary manual assessment to evaluate the solution of this issue? If possible specify measures and your goal - e.g. * Solution installable with basic linux system administration skills * User interface understandable by non developer curators |
| Actual evaluations | links to acutual evaluations of this Issue/Scenario |