Solving TIFF malformation using exiftool

Title

Solving TIFF malformation using exiftool

Detailed description

The issue page describes the problem as (essentially): TIFF files being unusable, despite being "validated" by tools like JHOVE.

Solution Champion

aamato

Corresponding Issue(s)

Relevant "issues" page.

TL;DR

• Detect:

• Output:

• Fix:

Solution details

A lot of time was spent investigating the details of the file structure, for the images which could not be opened; which ultimately was a dead-end.

Eventually, we realised that the images were claiming to be 16-bit greyscale, but in fact, were actualy 8-bit greyscale. (Which, in retrospect, should have been more obvious, based on the rare error messages tools would give us.)

This meant that detection/correction suddenly became a lot easier: check for a discrepancy between image dimension and bit-depth, and the actual file size. (See below.) Similarly, because the pixel data was actually correct, we can correct the images by setting the correct value for the "BitsPerSample" tag.

We used the exiftool tool to do the detection/correction, because it supports so many options on the command line. In this case, we can do the maths to check "expected file size" versus "ACTUAL file size"; and we can also re-write the header-value we want to change.

Caveats: This technique, as described, will only work with single-channel (i.e., greyscale) images. Exiftool supports a limited (but impressive) amount of command-line manipulation. Unfortunately, for colour images, I couldn't figure out how to get it to do the maths on "number of bits per pixel", since they're represented (textually) as the number of bits per channel (e.g., "8 8 8" vs. "24"). You could still apply the same techniques, and continue to use exiftool, but you'd probably just use it for metadata extraction, and wrap it in a shell/python/whatever script to do the analysis.

Important questions and takeaways (see slides below)

• What does "valid" mean for a file?
  • Well-formed?
  • Verified externally by a tool?
  • Matching a spec?
  • Internally consistent?

• What do we learn from this?
  • Don't always assume your vendors/digitisers are doing the job right.
  • Don't always assume that "successful validation" is meaningful. (Also: learn the limitations of your tools.)
  • The only thing better than double-checking is triple-checking.
  • KNOW WHAT YOU ARE "PRESERVING"!
  • (and: exiftool is pretty awesome)

Tool Registry Link

ExifTool

Evaluation

???

Slides from final day (images/PPT)

[]

This was the starting point:

This was where we want to be:

So what did we do to get there?

We moved one bit!

One bit makes a big difference:

Raising questions about "valid":

So how can we not have to do this again? We can use `exiftool` to evaluate the relevant properties of the TIFF images, and flag up an inconsistency:

And exiftool also lets us fix the problem:

What do we learn?

Also: Consequences of bit-rot can be severe!