Title
Solving TIFF malformation using exiftool
Detailed description
The issue page
describes the problem as (essentially): TIFF files being unusable, despite being "validated" by tools like JHOVE.
Solution Champion
Corresponding Issue(s)
TL;DR
- Detect:
- Output:
- Fix:
Solution details
A lot of time was spent investigating the details of the file structure, for the images which could not be opened; which ultimately was a dead-end.
Eventually, we realised that the images were claiming to be 16-bit greyscale, but in fact, were actualy 8-bit greyscale. (Which, in retrospect, should have been more obvious, based on the rare error messages tools would give us.)
This meant that detection/correction suddenly became a lot easier: check for a discrepancy between image dimension and bit-depth, and the actual file size. (See below.) Similarly, because the pixel data was actually correct, we can correct the images by setting the correct value for the "BitsPerSample" tag.
We used the exiftool tool to do the detection/correction, because it supports so many options on the command line. In this case, we can do the maths to check "expected file size" versus "ACTUAL file size"; and we can also re-write the header-value we want to change.
Caveats: This technique, as described, will only work with single-channel (i.e., greyscale) images. Exiftool supports a limited (but impressive) amount of command-line manipulation. Unfortunately, for colour images, I couldn't figure out how to get it to do the maths on "number of bits per pixel", since they're represented (textually) as the number of bits per channel (e.g., "8 8 8" vs. "24"). You could still apply the same techniques, and continue to use exiftool, but you'd probably just use it for metadata extraction, and wrap it in a shell/python/whatever script to do the analysis.
Important questions and takeaways (see slides below)
- What does "valid" mean for a file?
- Well-formed?
- Verified externally by a tool?
- Matching a spec?
- Internally consistent?
- What do we learn from this?
- Don't always assume your vendors/digitisers are doing the job right.
- Don't always assume that "successful validation" is meaningful. (Also: learn the limitations of your tools.)
- The only thing better than double-checking is triple-checking.
- KNOW WHAT YOU ARE "PRESERVING"!
- (and: exiftool is pretty awesome)
Tool Registry Link
Evaluation
Maurice de Rooij
We are going to incorporate this check in our QA process and request our vendors to do so as well. We will notify JHOVE developers as well to request them to incorporate this check.
Slides from final day (images/PPT)
This was the starting point:
This was where we want to be:
So what did we do to get there?
We moved one bit!
(I'd like to suggest this for "least work done to achieve a SPRUCE mashup solution" 
)
One bit makes a big difference:
Raising questions about "valid":
So how can we not have to do this again? We can use `exiftool` to evaluate the relevant properties of the TIFF images, and flag up an inconsistency:
And exiftool also lets us fix the problem:
What do we learn?
1 Comment
comments.show.hideSep 22, 2012
Gary McGath
Nicely done! Consider the lead JHOVE developer notified.
As far as incorporating it into JHOVE goes, though, I think it's too much of a special-case check. If a TIFF file uses a compressed encoding, the file size may legitimately be less than width * height * depth.
The files you're describing aren't actually well-formed, since the pixel encoding stream which is claimed to be 16-bit will run off the end of the file. JHOVE _could_ check, for uncompressed encodings, if the calculated size of the stream is within the file boundaries and doesn't overwrite any tags. I've already got a request to make sure no tag storage is doubled up, so a more general checking of storage allocation within a TIFF is a possibility.