Tiny Changes Can Fool AI
There has been much discussion more recently (and some not so recently) on how minute changes to images can fool the smartest neural nets. Sharif et al. showed how to fool a neural net into classifying a Reese Witherspoon photo as Russell Crowe by adding a groovy pair of technicolor zebra-striped wayfarer frames. If that’s all it takes, then maybe Clark Kent was onto something after all.
We had our own experience with this recently, with our multi-class DetectNet airplane-detection model. The model usually produces confidence scores over 0.95 for clear images of big planes like airliners, but it’s fooled by this image:
To the human eye, it’s very easy to pick out these airplanes because two of the three are only lightly covered by clouds, and the third is only partially obscured. But the DetectNet model cannot perform on cloudy imagery because the training data did not contain a lot of clouds. So the model missed one plane altogether and gave low confidence scores of
.23 (for light obscurity) and
.90 (for very partial cloud cover). An obvious solution to this issue is seek out cloudy images to train our model with, but that’s going to cost extra time and money. Instead, we can work with existing data by adding synthetic clouds to clear images.
There are other kinds of variation in satellite imagery, like off-nadir angle, time of day, and atmospheric conditions that also need to be addressed in the future, but we’ll focus on cloud cover for now.
Clouds in Satellite Imagery
I researched for examples of adding clouds to images, and I mainly found instructions on using graphics software to create clouds, but I need a way to do this with code so we can repeat the process thousands or millions of times during training.
One additional thing to consider before choosing a method is that most people are accustomed to looking at clouds from ground level on planet earth, not from satellites in space. So we should have a look at some clouds from satellite images and then decide how to proceed
As you can see, clouds actually look pretty much the same from space. File that under “Today I Learned”, and let’s go make some clouds…
Generating Clouds Programmatically
I found a great example of creating clouds in
C. From looking at the source code, I can see that the author created some white noise, and then progressively upsampled smaller and smaller parts of the image to the original image size and stacked the results on top of each other. The result is a fractal noise pattern that looks an awful lot like clouds.
Before I present my program in Python, here’s the algorithm in plain English:
- generate an NxN white noise image
r1(where N is half the image width)
- cut out the upper-left quadrant of
r1and store as
r2to the original image size, and store the result
- Multiply the pixel values of
- repeat steps 2, 3 and 4 on (
r8, …) to produce (
r16, …) until
- Sum all of
r4, etc. to produce your cloud pattern.
The algorithm order varies from the code, but the result should be the same.
I’ve reproduced this algorithm in Python upgraded the interpolation to bicubic because its 2017 and its a great time to be alive.
NOTE: The following code was exported from a
Jupyter Notebook (python 3.4.3)
using nbconvert, so be careful of Jupyter/IPython specifics such as
%matplotlib inline and semicolons to suppress output. These details improve the appearance of the notebook, but they might cause issues outside of Jupyter Notebook.
We can make the clouds more granular by increasing the number 2 in
power_range = range(2, ..., but this will also change the number of images that are summed together. I added the normalization step to make the code robust to that change. It might be possible to save a few cycles by using the formula for the sum of a geometric series, but I’ll leave that exercise for the reader.
Adding Clouds to Images
Now that we can generate cloud patterns, lets add one to an image. We want to leave the image more or less the same, but stick semi-transparent clouds on top of it. In order to do that, we’ll use PIL, the Python Image Library.
First, load a clear, non-cloudy image.
Then, we’ll convert the cloud pattern into a PIL format so it’s ready to be merged with another image. Last, we can merge the images together.
RGB RGB (768, 768) (768, 768)
We used the
Image.blend function to combine the images. The third parameter is the alpha value to use for blending. If alpha is 0.0, we’ll get the original image with no clouds. If it’s 1.0, we’ll just get clouds with nothing else. Let’s try varying the alpha channel and see what it looks like.
Now we can see a range of images with different alpha levels. For augmentation, we’ll want to avoid images that are so cloudy we can’t make out the objects. For this reason, we’ll set an upper bound of 0.65 on the alpha parameter. On the other hand, we don’t want to waste our time creating augmented images that don’t even look cloudy, so let’s set a minimum alpha value of 0.2.
Now let’s create an add_clouds function that we can use to augment images in the future. To speed things up a little, we’re dropping PIL from the augmentation process and just using pure numpy to add clouds to the image (I’m still using PIL to load the image). This function assumes we’re dealing with square images. If you want it to work with non-square images, you’ll have to modify the script to resize the clouds to match the image size (hint: you can use
scipy.misc.imresize again to do this).
Finally, let’s add some clouds to my favorite map projection.