Solving TIFF malformation using exiftool

Skip to end of metadata
Go to start of metadata
Title

Solving TIFF malformation using exiftool

Detailed description

The issue page describes the problem as (essentially): TIFF files being unusable, despite being "validated" by tools like JHOVE.

Solution Champion

aamato

Corresponding Issue(s)

Relevant "issues" page.

TL;DR
  • Detect:
  • Output:
  • Fix:
Solution details

A lot of time was spent investigating the details of the file structure, for the images which could not be opened; which ultimately was a dead-end.

Eventually, we realised that the images were claiming to be 16-bit greyscale, but in fact, were actualy 8-bit greyscale. (Which, in retrospect, should have been more obvious, based on the rare error messages tools would give us.)

This meant that detection/correction suddenly became a lot easier: check for a discrepancy between image dimension and bit-depth, and the actual file size. (See below.) Similarly, because the pixel data was actually correct, we can correct the images by setting the correct value for the "BitsPerSample" tag.

We used the exiftool tool to do the detection/correction, because it supports so many options on the command line. In this case, we can do the maths to check "expected file size" versus "ACTUAL file size"; and we can also re-write the header-value we want to change.

Caveats: This technique, as described, will only work with single-channel (i.e., greyscale) images. Exiftool supports a limited (but impressive) amount of command-line manipulation. Unfortunately, for colour images, I couldn't figure out how to get it to do the maths on "number of bits per pixel", since they're represented (textually) as the number of bits per channel (e.g., "8 8 8" vs. "24"). You could still apply the same techniques, and continue to use exiftool, but you'd probably just use it for metadata extraction, and wrap it in a shell/python/whatever script to do the analysis.

Important questions and takeaways (see slides below)
  • What does "valid" mean for a file?
    • Well-formed?
    • Verified externally by a tool?
    • Matching a spec?
    • Internally consistent?
  • What do we learn from this?
    • Don't always assume your vendors/digitisers are doing the job right.
    • Don't always assume that "successful validation" is meaningful. (Also: learn the limitations of your tools.)
    • The only thing better than double-checking is triple-checking.
    • KNOW WHAT YOU ARE "PRESERVING"!
    • (and: exiftool is pretty awesome)
Tool Registry Link

ExifTool

Evaluation

Maurice de Rooij
We are going to incorporate this check in our QA process and request our vendors to do so as well. We will notify JHOVE developers as well to request them to incorporate this check.

Slides from final day (images/PPT)

PowerPoint Slides

This was the starting point:

This was where we want to be:

So what did we do to get there?

We moved one bit!

(I'd like to suggest this for "least work done to achieve a SPRUCE mashup solution" )

One bit makes a big difference:

Raising questions about "valid":

So how can we not have to do this again? We can use `exiftool` to evaluate the relevant properties of the TIFF images, and flag up an inconsistency:

And exiftool also lets us fix the problem:

What do we learn?

Also: Consequences of (even single-)bit-rot can be subtle and severe!

Labels:
spruce_london spruce_london Delete
solution solution Delete
characterisation characterisation Delete
bit_rot_detection bit_rot_detection Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Sep 22, 2012

    Nicely done! Consider the lead JHOVE developer notified.

    As far as incorporating it into JHOVE goes, though, I think it's too much of a special-case check. If a TIFF file uses a compressed encoding, the file size may legitimately be less than width * height * depth.

    The files you're describing aren't actually well-formed, since the pixel encoding stream which is claimed to be 16-bit will run off the end of the file. JHOVE _could_ check, for uncompressed encodings, if the calculated size of the stream is within the file boundaries and doesn't overwrite any tags. I've already got a request to make sure no tag storage is doubled up, so a more general checking of storage allocation within a TIFF is a possibility.