Summarize Images

This is a corpus tool rather than a page tool. It walks the content stream of every page in every file you give it, records what each image declares about itself - filter, colorspace, component count, bits per component, pixel dimensions - and prints distributions across the whole set. Use it to find out what a body of documents actually contains before you write code that has to handle all of it.

Reading dictionaries versus decoding pixels

There are two ways to enumerate images, and they answer different questions.

The example takes the first route because it is auditing thousands of files. If you want the images themselves, or where they sit on the page, use the extractor instead - see image extraction.

Doing it

Images reach a page two ways, and both have to be handled:

for _, op := range *operations {
    switch {
    case op.Operand == "BI" && len(op.Params) == 1:
        // Inline image: the data is in the operator itself.
        iimg, ok := op.Params[0].(*contentstream.ContentStreamInlineImage)
        if !ok {
            continue
        }
        cs, _ := iimg.GetColorSpace(resources)
        encoder, _ := iimg.GetEncoder()

    case op.Operand == "Do" && len(op.Params) == 1:
        // Named XObject: could be an image or a form.
        name := op.Params[0].(*core.PdfObjectName)
        _, xtype := resources.GetXObjectByName(*name)
        if xtype == model.XObjectTypeImage {
            ximg, err := resources.GetXObjectImageByName(*name)
            // ximg.Filter, ximg.ColorSpace, ximg.Width, ximg.Height, ximg.BitsPerComponent
        }
    }
}

GetXObjectByName returns the stream and an XObjectType, one of XObjectTypeImage, XObjectTypeForm, XObjectTypePS, XObjectTypeUnknown or XObjectTypeUndefined. Checking it before calling GetXObjectImageByName matters, because a Do on a form XObject is how nested content gets drawn: the form has its own content stream and its own resources, and any images inside it are only found by recursing. contentStreamImages calls itself for that case, falling back to the parent page’s resources when the form declares none of its own.

ximg.Width, Height and BitsPerComponent are *int64 and can be nil, so guard before dereferencing. The example does, which is why it reports zeros rather than panicking on a malformed image dictionary.

Limitations

Each named XObject is processed once per content stream. A logo drawn ten times on a page counts as one image, which is what you want for an inventory and not what you want if you are counting draw operations.

Image dimensions are the pixel dimensions from the image dictionary, not the size the image occupies on the page. A 2000 pixel wide image scaled into a 100 point box reports 2000. Display geometry comes from the current transformation matrix, which this walk does not track; extractor.ExtractPageImages does.

ToImage() is still called, once per image, to get the component count. That decodes the image data, so the run is not as cheap as reading dictionaries alone; on a large corpus this dominates the time.

The corpus is capped at 1000 files. Anything beyond that is silently dropped, so pass files in batches if you have more.

Images referenced only from annotation appearance streams, and images in pages the reader fails to load, are not seen. Both cases log and continue rather than aborting the run.

Run the example

The example takes any number of PDF paths, sorts them by file size, and prints a summary grouped by inline flag, filter, colorspace, component count and bit depth, each as counts by image and by file. It also writes a row per unique image to results.csv. Start reading at contentStreamImages, which is where the actual detection happens; -o sets the CSV path, -p drops page numbers from the rows and -w drops dimensions.

git clone https://github.com/unidoc/unipdf-examples.git
cd unipdf-examples/analysis
go run pdf_summarize_images.go ~/testdata/*.pdf

If this is your first time using UniPDF, follow the getting started guide to create an API key and set up your development environment.

Sample output

Trimmed to two of the five distributions:

   0 of 1 "PDF32000_2008.pdf" 21.4 MB, 756 pages, 172 images, 1.1 sec
=================================================
Totals: 1 of 1 files contain images.    172 images
-----------------------------------------
filter
By image: 4
	      DCTDecode	   128 of 172 (74.4%)
	    FlateDecode	    39 of 172 (22.7%)
	 CCITTFaxDecode	     4 of 172 ( 2.3%)
	            Raw	     1 of 172 ( 0.6%)
By file: 4
	 CCITTFaxDecode	     1 of 1 (100.0%)
	      DCTDecode	     1 of 1 (100.0%)
	    FlateDecode	     1 of 1 (100.0%)
	            Raw	     1 of 1 (100.0%)
-----------------------------------------
bpc
By image: 2
	              8	   168 of 172 (97.7%)
	              1	     4 of 172 ( 2.3%)
By file: 2
	              1	     1 of 1 (100.0%)
	              8	     1 of 1 (100.0%)