View on GitHub

data-science

Notebooks and Python about data science

If you like this project please add your Star

Standard GAN on MNIST

Based on the original Generative Adversarial Network (GAN), as introduced by Goodfellow et al. in 2014 [1]

Learning goals

  • Learn about the GAN deep neural network
  • Design a clean implementation using Keras high level models (Sequential)
  • Use the new Tensorflow Dataset input data pipeline
In [1]:
COLAB = True
if COLAB:
  from google.colab import drive
  drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
In [2]:
!pip install tensorview
Requirement already satisfied: tensorview in /usr/local/lib/python3.6/dist-packages (0.4.1)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from tensorview) (3.2.2)
Requirement already satisfied: pyecharts-snapshot>=0.1.10tensorflow>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorview) (0.2.0)
Requirement already satisfied: pandas>=0.24.1 in /usr/local/lib/python3.6/dist-packages (from tensorview) (1.1.5)
Requirement already satisfied: pyecharts>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorview) (1.9.0)
Requirement already satisfied: linora>=0.9.3 in /usr/local/lib/python3.6/dist-packages (from tensorview) (0.9.3)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->tensorview) (2.4.7)
Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.6/dist-packages (from matplotlib->tensorview) (1.18.5)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->tensorview) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->tensorview) (2.8.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->tensorview) (1.3.1)
Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from pyecharts-snapshot>=0.1.10tensorflow>=2.0.0->tensorview) (7.0.0)
Requirement already satisfied: pyppeteer>=0.0.25 in /usr/local/lib/python3.6/dist-packages (from pyecharts-snapshot>=0.1.10tensorflow>=2.0.0->tensorview) (0.2.2)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.24.1->tensorview) (2018.9)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.6/dist-packages (from pyecharts>=1.2.0->tensorview) (2.11.2)
Requirement already satisfied: prettytable in /usr/local/lib/python3.6/dist-packages (from pyecharts>=1.2.0->tensorview) (2.0.0)
Requirement already satisfied: simplejson in /usr/local/lib/python3.6/dist-packages (from pyecharts>=1.2.0->tensorview) (3.17.2)
Requirement already satisfied: xgboost>=0.81 in /usr/local/lib/python3.6/dist-packages (from linora>=0.9.3->tensorview) (0.90)
Requirement already satisfied: tensorflow>=2.0.0rc0 in /usr/local/lib/python3.6/dist-packages (from linora>=0.9.3->tensorview) (2.3.0)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib->tensorview) (1.15.0)
Requirement already satisfied: urllib3<2.0.0,>=1.25.8 in /usr/local/lib/python3.6/dist-packages (from pyppeteer>=0.0.25->pyecharts-snapshot>=0.1.10tensorflow>=2.0.0->tensorview) (1.26.2)
Requirement already satisfied: appdirs<2.0.0,>=1.4.3 in /usr/local/lib/python3.6/dist-packages (from pyppeteer>=0.0.25->pyecharts-snapshot>=0.1.10tensorflow>=2.0.0->tensorview) (1.4.4)
Requirement already satisfied: tqdm<5.0.0,>=4.42.1 in /usr/local/lib/python3.6/dist-packages (from pyppeteer>=0.0.25->pyecharts-snapshot>=0.1.10tensorflow>=2.0.0->tensorview) (4.54.1)
Requirement already satisfied: websockets<9.0,>=8.1 in /usr/local/lib/python3.6/dist-packages (from pyppeteer>=0.0.25->pyecharts-snapshot>=0.1.10tensorflow>=2.0.0->tensorview) (8.1)
Requirement already satisfied: pyee<8.0.0,>=7.0.1 in /usr/local/lib/python3.6/dist-packages (from pyppeteer>=0.0.25->pyecharts-snapshot>=0.1.10tensorflow>=2.0.0->tensorview) (7.0.4)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from jinja2->pyecharts>=1.2.0->tensorview) (1.1.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from prettytable->pyecharts>=1.2.0->tensorview) (50.3.2)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from prettytable->pyecharts>=1.2.0->tensorview) (0.2.5)
Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from xgboost>=0.81->linora>=0.9.3->tensorview) (1.4.1)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (3.3.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (1.1.0)
Requirement already satisfied: tensorflow-estimator<2.4.0,>=2.3.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (2.3.0)
Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (1.12.1)
Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (0.10.0)
Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (2.10.0)
Requirement already satisfied: keras-preprocessing<1.2,>=1.1.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (1.1.2)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (0.36.1)
Requirement already satisfied: tensorboard<3,>=2.3.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (2.3.0)
Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (3.12.4)
Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (1.34.0)
Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (1.6.3)
Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (0.2.0)
Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (0.3.3)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (2.23.0)
Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (1.17.2)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (1.0.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (0.4.2)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (1.7.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (3.3.3)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (2020.12.5)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (2.10)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (4.1.1)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (4.6)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (1.3.0)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (3.1.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (3.1.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow>=2.0.0rc0->linora>=0.9.3->tensorview) (3.4.0)
In [3]:
import sys
import tensorflow as tf
import plotly.graph_objects as go
from ipywidgets import widgets
import numpy as np
from tensorflow.keras import models, layers, losses, optimizers, metrics
import tensorflow_datasets as tf_ds
import tensorview as tv
import matplotlib.pyplot as plt
from pathlib import Path
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.2) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
In [4]:
if COLAB:
  model_path = Path('/content/drive/My Drive/Colab Notebooks/DsStepByStep')
else:
  model_path = Path('model')
In [5]:
batch_size = 100
latent_dim = 100
image_width, image_height, image_channels = 32, 32, 1
mnist_dim = image_width * image_height * image_channels
In [6]:
disc_learning_rate = 0.0002
gen_learning_rate  = 0.0002

relu_alpha = 0.01

Data

MNIST dataset is optimized to be stored efficiently: images are closely cropped at 28x28 pixels and stored as 1 byte per pixel (uint8 format). However, to get proper performance we need to modify the input data to insert some padding around and convert the pixel format to float on 32 bits.

In [7]:
def normalize_img(image, label):
  """Normalizes images: `uint8` -> `float32` and pad to 32 x 32"""
  image_float = tf.cast(image, tf.float32) / 128. - 1.
  image_padded = tf.pad(image_float, [[0, 0], [2, 2], [2, 2], [0, 0]])
  return image_padded, label

(ds_train, ds_test) = tf_ds.load('mnist', split=['train', 'test'], batch_size=batch_size, as_supervised=True)
ds_train = ds_train.map(normalize_img)
ds_train = ds_train.cache()

ds_test = ds_test.map(normalize_img)
ds_test = ds_test.cache()

ds_train, ds_test
Downloading and preparing dataset mnist/3.0.1 (download: 11.06 MiB, generated: 21.00 MiB, total: 32.06 MiB) to /root/tensorflow_datasets/mnist/3.0.1...
WARNING:absl:Dataset mnist is hosted on GCS. It will automatically be downloaded to your
local data directory. If you'd instead prefer to read directly from our public
GCS bucket (recommended if you're running on GCP), you can instead pass
`try_gcs=True` to `tfds.load` or set `data_dir=gs://tfds-data/datasets`.


Dataset mnist downloaded and prepared to /root/tensorflow_datasets/mnist/3.0.1. Subsequent calls will reuse this data.
Out[7]:
(<CacheDataset shapes: ((None, 32, 32, 1), (None,)), types: (tf.float32, tf.int64)>,
 <CacheDataset shapes: ((None, 32, 32, 1), (None,)), types: (tf.float32, tf.int64)>)

Models

GAN model is built out of a generator and a discriminator:

  • The generator gets as input some random noise on space of 100 dimensions, and issues an image (32x32 pixel raster)
  • The discriminator is trained to distinguish generated images by the generator (i.e. fakes), and reference images from the MNIST

The generator and discriminator architecture are more or less symmetrical. The generator is increasing the output space dimension step by step using wider and wider layers. The discriminator is similar to other classification networks reducing the input space dimensions down to the binary classification layer.

The "game" is to jointly train the generator and discriminator in order to have the best generator but still being able to detect generated images.

In [8]:
generator = models.Sequential([
    layers.Dense(256, input_dim=latent_dim),
    layers.LeakyReLU(relu_alpha),
    layers.Dropout(0.3),
    layers.Dense(512),
    layers.LeakyReLU(relu_alpha),
    layers.Dense(1024),
    layers.LeakyReLU(relu_alpha),
    layers.Dense(mnist_dim, activation='tanh'),
    layers.Reshape([32, 32, 1]),
], name='generator')

generator.compile()
generator.summary()
Model: "generator"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 256)               25856     
_________________________________________________________________
leaky_re_lu (LeakyReLU)      (None, 256)               0         
_________________________________________________________________
dropout (Dropout)            (None, 256)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 512)               131584    
_________________________________________________________________
leaky_re_lu_1 (LeakyReLU)    (None, 512)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 1024)              525312    
_________________________________________________________________
leaky_re_lu_2 (LeakyReLU)    (None, 1024)              0         
_________________________________________________________________
dense_3 (Dense)              (None, 1024)              1049600   
_________________________________________________________________
reshape (Reshape)            (None, 32, 32, 1)         0         
=================================================================
Total params: 1,732,352
Trainable params: 1,732,352
Non-trainable params: 0
_________________________________________________________________
In [9]:
discriminator = models.Sequential([
    layers.Input(shape=[32, 32, 1]),
    layers.Flatten(),
    layers.Dropout(0.3),
    layers.Dense(1024),
    layers.LeakyReLU(relu_alpha),
    layers.Dense(512),
    layers.LeakyReLU(),
    layers.Dense(256),
    layers.LeakyReLU(relu_alpha),
    layers.Dense(1) # activation='sigmoid'
], name='discriminator')

discriminator.compile()
discriminator.summary()
Model: "discriminator"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
flatten (Flatten)            (None, 1024)              0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 1024)              0         
_________________________________________________________________
dense_4 (Dense)              (None, 1024)              1049600   
_________________________________________________________________
leaky_re_lu_3 (LeakyReLU)    (None, 1024)              0         
_________________________________________________________________
dense_5 (Dense)              (None, 512)               524800    
_________________________________________________________________
leaky_re_lu_4 (LeakyReLU)    (None, 512)               0         
_________________________________________________________________
dense_6 (Dense)              (None, 256)               131328    
_________________________________________________________________
leaky_re_lu_5 (LeakyReLU)    (None, 256)               0         
_________________________________________________________________
dense_7 (Dense)              (None, 1)                 257       
=================================================================
Total params: 1,705,985
Trainable params: 1,705,985
Non-trainable params: 0
_________________________________________________________________

Training

Training is alternatively on the distriminator and generator.

The discriminator is trained on a batch made of half genuine images and half trained images.

The generator is trained with its output fed into the discriminator (whose wheights are frozen in this phase).

GAN reputation as difficult to be trained is well deserved and originates in the joint optimization which is similar to a minimax problem (min discrination error, max fidelity of the fakes). As seen below, the noise on the losses and accuracies is high. The main facilitators helping this training are:

  • Use of leaky ReLU activations to avoid gradient vanishing
  • Small learning rate to decrease the noise and instability
In [10]:
epochs = 60
batch_per_epoch = 60000/batch_size
loss_object = tf.keras.losses.BinaryCrossentropy(from_logits=True)
In [11]:
def generator_loss(disc_generated_output):
    return loss_object(tf.ones_like(disc_generated_output), disc_generated_output)
In [12]:
def discriminator_loss(disc_real_output, disc_generated_output):

    real_loss = loss_object(tf.ones_like(disc_real_output), disc_real_output)
    generated_loss = loss_object(tf.zeros_like(disc_generated_output), disc_generated_output)

    return real_loss + generated_loss
In [13]:
@tf.function
def train_step(generator, discriminator, 
               generator_optimizer, discriminator_optimizer, 
               generator_latent, batch, 
               epoch):
    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
        
        gen_latent = generator_latent()
        
        gen_output = generator(gen_latent, training=True)

        disc_real_output = discriminator(batch, training=True)
        disc_generated_output = discriminator(gen_output, training=True)

        gen_loss = generator_loss(disc_generated_output)
        disc_loss = discriminator_loss(disc_real_output, disc_generated_output)

        generator_gradients = gen_tape.gradient(gen_loss, generator.trainable_variables)
        discriminator_gradients = disc_tape.gradient(disc_loss, discriminator.trainable_variables)

    generator_optimizer.apply_gradients(zip(generator_gradients, generator.trainable_variables))
    discriminator_optimizer.apply_gradients(zip(discriminator_gradients, discriminator.trainable_variables))

    return gen_loss, disc_loss
In [14]:
generator_optimizer = tf.keras.optimizers.Adam(gen_learning_rate, beta_1=0.05)
discriminator_optimizer = tf.keras.optimizers.Adam(disc_learning_rate, beta_1=0.05)
In [15]:
tv_plot = tv.train.PlotMetrics(wait_num=200, columns=2, iter_num=epochs * batch_per_epoch)

def generator_latent():
    return tf.random.normal((batch_size, latent_dim), 0, 1)

for epoch in range(epochs):

    for train_batch in iter(ds_train):
        
        g_loss, d_loss = train_step(generator, discriminator, 
                                    generator_optimizer, discriminator_optimizer, 
                                    generator_latent, train_batch[0], 
                                    epoch)
        # Plot
        tv_plot.update({ 'discriminator_loss': d_loss,# 'discriminator_acc': d_acc,
                        'generator_loss': g_loss, # 'generator_acc': g_acc
                       })
        tv_plot.draw()
In [16]:
gen_latent = generator_latent()
gen_imgs = generator(gen_latent, training=True).numpy()

fig, axes = plt.subplots(8, 8, sharex=True, sharey=True, figsize=(10, 10))
for img, ax in zip(gen_imgs, axes.ravel()):
    ax.imshow(img.reshape(image_width, image_height), interpolation='nearest', cmap='gray')
    ax.axis('off')
fig.tight_layout()
In [17]:
discriminator.save(model_path / 'mnist_gan_discriminator.h5')
generator.save(model_path / 'mnist_gan_generator.h5')

Conclusion

The generator is not fooling that much the discriminator since the discriminator accuracy is well above 50% and its loss is stable at a low level. However, the generated digits are quite well looking to an human eye.

Where to go from here

  • More advanced GAN based on convolutional layers (HTML / Jupyter)
  • Revisit the fundamentals about deep neural networks in the CNN versus Dense classification (HTML / Jupyter )

References

  1. "Generative adversarial nets", I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, NIPS 2014