Creating Synthetic Clouds in Python

By: Alan J. Schoen
Data Scientist

Tiny Changes Can Fool AI

There has been much discussion more recently (and some not so recently) on how minute changes to images can fool the smartest neural nets. Sharif et al. showed how to fool a neural net into classifying a Reese Witherspoon photo as Russell Crowe by adding a groovy pair of technicolor zebra-striped wayfarer frames. If that’s all it takes, then maybe Clark Kent was onto something after all.

We had our own experience with this recently, with our multi-class DetectNet airplane-detection model. The model usually produces confidence scores over 0.95 for clear images of big planes like airliners, but it’s fooled by this image:

A cloudy image Detection problems

To the human eye, it’s very easy to pick out these airplanes because two of the three are only lightly covered by clouds, and the third is only partially obscured. But the DetectNet model cannot perform on cloudy imagery because the training data did not contain a lot of clouds. So the model missed one plane altogether and gave low confidence scores of .23 (for light obscurity) and .90 (for very partial cloud cover). An obvious solution to this issue is seek out cloudy images to train our model with, but that’s going to cost extra time and money. Instead, we can work with existing data by adding synthetic clouds to clear images.

There are other kinds of variation in satellite imagery, like off-nadir angle, time of day, and atmospheric conditions that also need to be addressed in the future, but we’ll focus on cloud cover for now.

Clouds in Satellite Imagery

I researched for examples of adding clouds to images, and I mainly found instructions on using graphics software to create clouds, but I need a way to do this with code so we can repeat the process thousands or millions of times during training.

One additional thing to consider before choosing a method is that most people are accustomed to looking at clouds from ground level on planet earth, not from satellites in space. So we should have a look at some clouds from satellite images and then decide how to proceed

Satellite Clouds Satellite Clouds Satellite Clouds Satellite Clouds

As you can see, clouds actually look pretty much the same from space. File that under “Today I Learned”, and let’s go make some clouds…

Generating Clouds Programmatically

I found a great example of creating clouds in C. From looking at the source code, I can see that the author created some white noise, and then progressively upsampled smaller and smaller parts of the image to the original image size and stacked the results on top of each other. The result is a fractal noise pattern that looks an awful lot like clouds.

Before I present my program in Python, here’s the algorithm in plain English:

  1. generate an NxN white noise image r1 (where N is half the image width)
  2. cut out the upper-left quadrant of r1 and store as r2
  3. upsample r2 to the original image size, and store the result
  4. Multiply the pixel values of r2 by 2
  5. repeat steps 2, 3 and 4 on (r2, r4, r8, …) to produce (r4, r8, r16, …) until rN is 1x1
  6. Sum all of r1, r2, r4, etc. to produce your cloud pattern.

The algorithm order varies from the code, but the result should be the same.

I’ve reproduced this algorithm in Python upgraded the interpolation to bicubic because its 2017 and its a great time to be alive.

NOTE: The following code was exported from a Jupyter Notebook (python 3.4.3) using nbconvert, so be careful of Jupyter/IPython specifics such as %matplotlib inline and semicolons to suppress output. These details improve the appearance of the notebook, but they might cause issues outside of Jupyter Notebook.

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import scipy
import scipy.ndimage
%matplotlib inline

# Define the make_turbulence function, which will create noise patterns
def make_turbulence(im_size):
    # Initialize the white noise pattern
    base_pattern = np.random.uniform(0,255, (im_size//2, im_size//2))

    # Initialize the output pattern
    turbulence_pattern = np.zeros((im_size, im_size))

    # Create cloud pattern
    power_range = range(2, int(np.log2(im_size)))
    for i in power_range:

        # Set the size of the quadrant to work on
        subimg_size = 2**i

        # Extract the pixels in the upper left quadrant
        quadrant = base_pattern[:subimg_size, :subimg_size]

        # Up-sample the quadrant to the original image size
        # intperp can be 'nearest', 'lanczos', 'bilinear', 'bicubic' or 'cubic'
        upsampled_pattern = scipy.misc.imresize(quadrant, (im_size, im_size), interp='bicubic')
        
        # Add the new noise pattern to the result
        turbulence_pattern += upsampled_pattern / subimg_size

    # Normalize values
    turbulence_pattern /= sum([1 / 2**i for i in power_range])
    
    return turbulence_pattern

# Generate a cloud pattern
turbulence_pattern = make_turbulence(768)

# Plot the results
_, ax = plt.subplots(figsize=(12, 12))
norm = matplotlib.colors.Normalize(vmin=0, vmax=255, clip=True)
ax.imshow(turbulence_pattern, cmap=plt.cm.gray, norm=norm)
ax.axis('off');

Synthetic Clouds

We can make the clouds more granular by increasing the number 2 in power_range = range(2, ..., but this will also change the number of images that are summed together. I added the normalization step to make the code robust to that change. It might be possible to save a few cycles by using the formula for the sum of a geometric series, but I’ll leave that exercise for the reader.

Adding Clouds to Images

Now that we can generate cloud patterns, lets add one to an image. We want to leave the image more or less the same, but stick semi-transparent clouds on top of it. In order to do that, we’ll use PIL, the Python Image Library.

First, load a clear, non-cloudy image.

from PIL import Image

# Load the clear image
img = Image.open('61.png')

img

png

Then, we’ll convert the cloud pattern into a PIL format so it’s ready to be merged with another image. Last, we can merge the images together.

# Convert the turbulence pattern into a PIL RGB image
turb_img = Image.fromarray(np.dstack([turbulence_pattern.astype(np.uint8)]*3))

# Make sure that the images match each other
print(img.mode, turb_img.mode)
print(img.size, turb_img.size)

# Now try blending them
Image.blend(img, turb_img, 0.5)
RGB RGB
(768, 768) (768, 768)

png

We used the Image.blend function to combine the images. The third parameter is the alpha value to use for blending. If alpha is 0.0, we’ll get the original image with no clouds. If it’s 1.0, we’ll just get clouds with nothing else. Let’s try varying the alpha channel and see what it looks like.

# Initialize a grid of subplots 
n_levels = 12
fig, axes = plt.subplots(3, n_levels//3, figsize=(20,16))

# Iterate through each alpha value, plottig the results
flag = True
for ax, alpha in zip(axes.flat, np.linspace(0,1,n_levels)):
    ax.imshow(Image.blend(img, turb_img, alpha))
    ax.axis('off')
    if flag:
        ax.set_title("alpha:    {:.2f}             ".format(alpha), fontsize=20, color='gray')
        flag = False
    else:
        ax.set_title("{:.2f}".format(alpha), fontsize=20, color='gray')

png

Now we can see a range of images with different alpha levels. For augmentation, we’ll want to avoid images that are so cloudy we can’t make out the objects. For this reason, we’ll set an upper bound of 0.65 on the alpha parameter. On the other hand, we don’t want to waste our time creating augmented images that don’t even look cloudy, so let’s set a minimum alpha value of 0.2.

Now let’s create an add_clouds function that we can use to augment images in the future. To speed things up a little, we’re dropping PIL from the augmentation process and just using pure numpy to add clouds to the image (I’m still using PIL to load the image). This function assumes we’re dealing with square images. If you want it to work with non-square images, you’ll have to modify the script to resize the clouds to match the image size (hint: you can use scipy.misc.imresize again to do this).

Finally, let’s add some clouds to my favorite map projection.

def add_clouds(image_np, alpha):
    turbulence_pattern = make_turbulence(image_np.shape[0])
    return (image_np*(1-alpha) + turbulence_pattern[:,:,None]*alpha).astype(np.uint8)

image_np = np.array(Image.open('Peirce_quincuncial_projection_SW_20W.JPG'))
alpha = np.random.uniform(0.2, 0.65)
cloudy_image_np = add_clouds(image_np, alpha)

_, ax = plt.subplots(figsize=(12,12))
ax.imshow(cloudy_image_np)
ax.axis('off');

png

You can download my Jupyter notebook here.
If you’d like to view it rendered in GitHub, try this link.

comments powered by Disqus

Contact the DeepCore team!

Questions, comments, or requests? Contact the DeepCore team for more information!