Fancy transforms

The development libvips, 8.2, has just gained a nice new feature: a fancy transformation operation.

The idea is that you make an index image where each pixel is a pair of numbers. These numbers are the coordinates of a point in another image. The transform operation, vips_mapim(), takes the index image and a source image and generates the output by looking up every point in the index in the input. This is a pretty standard feature of image processing libraries and libvips should have had it years ago, ah well.

Here's a simple Python example to show this in action. It uses vips_xyz() to make an image where each pixel is its own coordinate, then uses pixel arithmetic to displace pixels by sin(distance) / distance. When you apply this index image to a photo, it gives a wobble effect that fades to the edges of the image.

#!/usr/bin/python

import sys

import gi
gi.require_version('Vips', '8.0')
from gi.repository import Vips

def wobble(image):
    # this makes an image where pixel (0, 0) (at the top-left) has value [0, 0],
    # and pixel (image.width, image.height) at the bottom-right has value
    # [image.width, image.height]
    index = Vips.Image.xyz(image.width, image.height)

    # make a version with (0, 0) at the centre, negative values up and left,
    # positive down and right
    centre = index - [image.width / 2, image.height / 2]

    # to polar space, so each pixel is now distance and angle in degrees
    polar = centre.polar()

    # scale sin(distance) by 1/distance to make a wavey pattern
    d = 10000 * (polar[0] * 3).sin() / (1 + polar[0])

    # and back to rectangular coordinates again to make a set of vectors we can
    # apply to the original index image
    index += d.ibandjoin(polar[1]).rect()

    # finally, use our modified index image to distort the input!
    return image.mapim(index)

image = Vips.Image.new_from_file(sys.argv[1])
image = wobble(image)
image.write_to_file(sys.argv[2])

Here's what the Bilbao Guggenheim looks like through this program:

This is doing the complete calculation for every pixel, but often that level of accuracy is not really necessary. We could generate a quarter-resolution index and scale it up, then we'd only need to do 1/16th of the maths.

This is a simple change to make to the program. We swap the xyz line for:

    index = 4 * Vips.Image.xyz(image.width / 4, image.height / 4)

that is, generate 1/4 of the image, but scale values up by 4. And we change the way we apply the index:

    return image.mapim(index.similarity(scale = 4))

Now we enlarge the index to full size before doing the lookup.

It won't make much difference on a small image like this, but on a larger file it starts to become noticeable. With a 10,000 by 10,000 pixel jpeg I see:

$ time ./wobble.py ~/pics/wtc.jpg x.jpg
memory: high-water mark 56.43 MB
real    0m7.644s
user    0m14.320s
sys    0m0.517s
$ time ./wobble4.py ~/pics/wtc.jpg x.jpg
memory: high-water mark 45.24 MB
real    0m5.205s
user    0m7.676s
sys    0m0.413s

A nice improvement. For comparison, imagemagick's swirl operation, which should be roughly comparable, takes about 3x longer and needs a lot more memory:

$ time convert ~/pics/wtc.jpg -swirl 1080 x.jpg
peak memory: 1.5 GB
real    0m16.746s
user    0m16.079s
sys    0m0.621s

nip2 is nice for playing about with this stuff, you can watch the index image change as you adjust equations. Here's nip2 doing the same thing, with some sliders to adjust the constants:

Workspace plus sample image here.

You can use vips_mapim() for useful things too. Here are a pair of functions which map images to and from polar coordinate space:

def to_polar(image):
    xy = Vips.Image.xyz(image.width, image.height)
    xy -= [image.width / 2.0, image.height / 2.0]
    scale = min(image.width, image.height) / float(image.width)
    xy *= 2.0 / scale
    index = xy.polar()
    index *= [1, image.height / 360.0]
    return image.mapim(index)

def to_rectangular(image):
    xy = Vips.Image.xyz(image.width, image.height)
    xy *= [1, 360.0 / image.height]
    index = xy.rect()
    scale = min(image.width, image.height) / float(image.width)
    index *= scale / 2.0
    index += [image.width / 2.0, image.height / 2.0]
    return image.mapim(index)

Converting to polar wraps an image around a vertical axis positioned at the origin. With the Guggenheim image you get this strange thing:

This has the nice property that vertical lines in the input become circles (or segments of circles) in polar space, and horizontal lines become radial spokes.

Here's an example of using this transform taken from this stackoverflow answer. This image is a scanned page from a book, but it's annoyingly not quite straight:

If we take the FFT of this, you can see the angle of the page as an angled line in the transform, caused by the recurring shapes of characters:

Now, if we turn this to rectangular coordinates, we'll see radial lines as horizontals:

Now just sum each row to get a set of peaks:

And the position of the largest peak is the angle the image is rotated by. Applying that rotation to the original gives:

which looks pretty straight. There's a nip2 workspace plus a sample image if you'd like to try it out. You'll need git master libvips and nip2.