Solving TIFF malformation using exiftool
The issue page describes the problem as (essentially): TIFF files being unusable, despite being "validated" by tools like JHOVE.
A lot of time was spent investigating the details of the file structure, for the images which could not be opened; which ultimately was a dead-end.
Eventually, we realised that the images were claiming to be 16-bit greyscale, but in fact, were actualy 8-bit greyscale. (Which, in retrospect, should have been more obvious, based on the rare error messages tools would give us.)
This meant that detection/correction suddenly became a lot easier: check for a discrepancy between image dimension and bit-depth, and the actual file size. (See below.) Similarly, because the pixel data was actually correct, we can correct the images by setting the correct value for the "BitsPerSample" tag.
We used the exiftool tool to do the detection/correction, because it supports so many options on the command line. In this case, we can do the maths to check "expected file size" versus "ACTUAL file size"; and we can also re-write the header-value we want to change.
Caveats: This technique, as described, will only work with single-channel (i.e., greyscale) images. Exiftool supports a limited (but impressive) amount of command-line manipulation. Unfortunately, for colour images, I couldn't figure out how to get it to do the maths on "number of bits per pixel", since they're represented (textually) as the number of bits per channel (e.g., "8 8 8" vs. "24"). You could still apply the same techniques, and continue to use exiftool, but you'd probably just use it for metadata extraction, and wrap it in a shell/python/whatever script to do the analysis.
- What does "valid" mean for a file?
- Verified externally by a tool?
- Matching a spec?
- Internally consistent?
- What do we learn from this?
- Don't always assume your vendors/digitisers are doing the job right.
- Don't always assume that "successful validation" is meaningful. (Also: learn the limitations of your tools.)
- The only thing better than double-checking is triple-checking.
- KNOW WHAT YOU ARE "PRESERVING"!
- (and: exiftool is pretty awesome)
Maurice de Rooij
We are going to incorporate this check in our QA process and request our vendors to do so as well. We will notify JHOVE developers as well to request them to incorporate this check.
(I'd like to suggest this for "least work done to achieve a SPRUCE mashup solution" )
So how can we not have to do this again? We can use `exiftool` to evaluate the relevant properties of the TIFF images, and flag up an inconsistency: