Essays, Posts & Presentations

Generative Adversarial Networks (GANs) in the Wolfram Language

Jérôme Louradour

Author

Jérôme Louradour

Title

Generative Adversarial Networks (GANs) in the Wolfram Language

Description

New GAN training functions in the Wolfram Language and how to use them. Wrap a generator and a discriminator into a training network that computes the losses. Apply a custom update schedule.

Generative Adversarial Networks (GANs)
in the Wolfram Language

August 18, 2020

Jérôme Louradour, Machine Learning

A noteworthy achievement of artificial intelligence, since it is driven by artificial neural networks under the label deep learning, is the ability to create artistic works to generate images, text and sounds. At the core of this breakthrough is a basic method to train neural networks that was introduced by Ian Goodfellow in 2014 and was called by Yann LeCun “the most interesting idea in the last 10 years in machine learning”: generative adversarial networks (GANs). A GAN is a way to train a generative network that produces realistic-looking fake samples out of a latent seed, which can be some arbitrary data or random numbers sampled from a simple distribution. Let’s look at how to do so with some of the new capabilities developed for Mathematica Version 12.1.

Adversarial training found many applications, particularly in image processing: photo editing, style transfer, colorization, inpainting, super resolution, generation of images from a text, etc. It can also improve the accuracy of image recognition models by augmenting the data to train them. GANs can also be used just for fun. Look how these networks available in the Wolfram Neural Net Repository are able to change apples into oranges:

In[]:=

NetModel["CycleGAN Apple-to-Orange Translation Trained on ImageNet Competition Data"]



Out[]=

Contents cannot be rendered at this time; please try again later

Just for fun, we tested the networks by giving them a silly-looking partial horse picture, and they passed! The image was still recognized as a horse, and we were able to change it into a zebra:

In[]:=

NetModel["CycleGAN Horse-to-Zebra Translation Trained on ImageNet Competition Data"]

Contents cannot be rendered at this time; please try again later



Out[]=

Contents cannot be rendered at this time; please try again later

Artificial Neural Networks Before diving into the details of GANs, let’s say some words about neural networks, which are at the core of deep learning. A neural network is simply a function with a lot of parameters that can be trained by gradient descent to minimize a loss. The function is usually expressed as a composition of basic functions, called layers. Neural networks are modular and able to model an infinity of functions by combining different layers, just as you can build an infinity of LEGO constructions by stacking building blocks. Their flexibility explains a part of their success.

The optimization of neural networks is now well mastered since machine learning has been democratized with user-friendly optimization toolboxes, such as

NetTrain

in the Wolfram Language. The main technical challenges that remain when implementing a deep learning approach for a given application, besides collecting the relevant data, consist of:

Designing the neural architecture (number of parameters, connectivity, ...)

Choosing a sensible loss function

GANs address the latter. They provide a good strategy to build a loss suitable for generating realistic-looking data, without having to model the underlying distribution of real data.

GANs in a Nutshell The principle of GANs is that the loss for the generative network is given by another network: the discriminative network. These two networks play a zero-sum game, like in tug-of-war.

The “discriminator” learns to distinguish fake samples from real ones, and the “generator” learns to fool the discriminator with fake samples that look real. The generator yields a fake sample by simply applying a forward pass to a latent seed that can be an array of random numbers or some arbitrary data.

The crux in a GAN is finding an equilibrium. If the discriminator starts to perform too poorly in comparison to the generator, it will be fooled forever and the generator will stop learning anything, and vice versa. It is like when one opponent in a tug-of-war game lets the rope go: he loses the game, and nothing prevents the other from falling.

Besides, because GAN losses are artificial and constantly changing over the training as the discriminator changes, none of the losses that are optimized are good indicators of whether the generator is doing a good job or not. Only extreme loss values can tell that GAN training is doomed. Finding good metrics to monitor GAN training and evaluate generated samples is an open question whose answer highly depends on the application.

The following chart depicts how the discriminator and the generator are applied:

In[]:=

Contents cannot be rendered at this time; please try again later

The adversarial training procedure of this pair of networks and losses consists of iteratively looping over the following steps after a random (or pretrained) initialization of the two networks:

◼

Discriminator update:

1) Sample a batch of

real data

2) Sample

latent seeds, and generate

fakes

3) Update the discriminator to minimize "LossReal" and "LossFake"

◼

Generator update:

4) Sample

latent seeds, and generate

fakes

5) Update the generator to maximize "LossFake"

The generator and the discriminator are never updated at the same time, which is actually crucial for good convergence behavior. Several forms of loss are possible, and the most popular ones are:

◼

The Jensen–Shannon losses, as proposed by Goodfellow et al. in “Generative Adversarial Nets” (2014), which is a log-likelihood and assumes that the discriminator outputs a probability:

Out[]=

LossReal=

batch size

∑

i=1

log(discriminator(sample(i)))

LossFake=

batch size

∑

i=1

log(1-discriminator(generator(latent(i))))

◼

The Wasserstein loss, as proposed by Martin Arjovsky et al. in “Wasserstein GAN” (2017), which aims to maximize the gap between scores assigned by the discriminator to real and fake examples, respectively:

LossReal=

batch size

∑

i=1

discriminator(sample(i))

LossFake=

batch size

∑

i=1

-discriminator(generator(latent(i)))

GANs in the Wolfram Language The design of GANs in the Wolfram Language took a lot of brainstorming sessions. For a long time, there was hesitation between using a simple-to-use and high-level interface based on NetTrain, or a more flexible but lower-level solution where machine learning experts could write custom training loops and run them efficiently. Developers finally opted for the former (a user-friendly design in Version 12.1), while continuing to work on more flexible and complicated syntaxes as well. I am now happy to demonstrate how easy it can be to write GAN experiments in the Wolfram Language’s high-level symbolic neural network framework.

In the end, we can unlock GAN training by introducing only two new symbols:

◼

NetGANOperator

, an operator that wraps a generator and a discriminator into a training network that computes the losses and that can be given directly as a first argument to NetTrain

◼

TrainingUpdateSchedule

, an option to NetTrain to apply a custom update schedule, as in the training loop of the aforementioned pseudo-code

The options

TrainingProgressFunction

and

TrainingProgressReporting

are also very useful to implement custom monitoring of the generator performance, like showing a compilation of generated samples at regular steps of the training.

LearningRateMultipliers

can also be useful to invert or negate gradients, which can unlock more customized versions than vanilla GANs. This option was revisited in Version 12.1 so that learning rate multipliers can be directly attached to any learnable array, layer or network. It is the first option for the training that is introduced in layers and networks. For many applications, it is actually quite handy to attach learning rate multipliers while building a network, rather than to have to attach them to network subparts at each call of NetTrain.

How NetGANOperator Works Let’s build a NetGANOperator and understand what is going inside this operator.

First, a generator needs to be defined. Like this deep neural network, which takes a vector of one hundred random values and produces an RGB image of size 224×224, reusing pretrained layers from an existing CycleGAN model:

In[]:=

pretrained=NetTake[NetModel["CycleGAN Horse-to-Zebra Translation Trained on ImageNet Competition Data"],{10,All}];generator=NetChain[{LinearLayer["Input"100],pretrained,ResizeLayer[{224,224}]},"Output"NetDecoder["Image"]]

Out[]=

NetChain



uniniti

aliz

	Input	vector (size: 100)
1	LinearLayer	array (size: 128×64×64)
2	NetChain (22 nodes)	array (size: 3×256×256)
3	ResizeLayer	array (size: 3×224×224)
	Output	image

FE`NeuralNetworks`Private`NetChain`selected$$9752633980601357857883117908161310740561

FE`NeuralNetworks`Private`NetChain`part$$9752633980601357857883117908161310740561



Only the first layer of this generator has to be initialized, and it is ready to produce images out of random numbers sampled from a simple distribution:

In[]:=

generator=NetInitialize[generator]

Out[]=

NetChain



	Input	vector (size: 100)
1	LinearLayer	array (size: 128×64×64)
2	NetChain (22 nodes)	array (size: 3×256×256)
3	ResizeLayer	array (size: 3×224×224)
	Output	image

FE`NeuralNetworks`Private`NetChain`selected$$8020357496096597221612504750136182887847

FE`NeuralNetworks`Private`NetChain`part$$8020357496096597221612504750136182887847



In[]:=

generator[RandomVariate[NormalDistribution[],100]]

Out[]=

Next, a discriminator is needed, like this deep neural network adapted from a popular image classifier:

In[]:=

discriminator=NetChain[{NetModel["ResNet-101 Trained on ImageNet Competition Data"],AggregationLayer[Max,1]}]

Out[]=

NetChain



Input

image

array

(size: 3×224×224)

NetChain (42 nodes)

vector

(size: 1000)

AggregationLayer

real

Output

real

FE`NeuralNetworks`Private`NetChain`selected$$8372476038225203688322956471245350433342

FE`NeuralNetworks`Private`NetChain`part$$8372476038225203688322956471245350433342



This discriminator is a binary classifier that produces the probability of an input RGB image of size 224×224 to be real, i.e. not faked by the generator:

In[]:=

AssociationMapdiscriminator,



Out[]=



0.263017,

0.794715

With a generator and a discriminator on hand, we are now ready to combine the two nets using NetGANOperator:

In[]:=

gan=NetGANOperator[{generator,discriminator},"JensenShannon"]

Out[]=

NetGANOperator



Parameters
Generator :	NetChain (24 nodes)
Discriminator :	NetChain (43 nodes)
Loss :	JensenShannon
Input Ports
Sample :	image
Latent :	vector (size: 100)
Output Ports
LossReal :	real
LossFake :	real
GeneratedFake :	image

FE`NeuralNetworks`Private`FormattingLayers`part$$8452060474883880543747295735012502381499



This operator takes two inputs: latent random numbers and real images. It computes generated images by applying the generator to the latent seeds as well as to the loss values on the discriminator’s outputs, respectively, for the generated images and the real ones:

In[]:=

gan<|"Sample"

,"Latent"NumericArray

Type: Real32

Dimensions: {100}

Data: 0.672, -0.631, 0.975, -0.196, …

|>

Out[]=

LossReal0.229772,LossFake0.0761646,GeneratedFake



NetGANOperator is not just a generative network, but a training network; it is ready to be passed directly as a first argument to NetTrain, and the generator can be easily extracted from it. NetGANOperator can also be combined into a

NetGraph

to build a more complex training network for approaches like CycleGAN, for instance. CycleGAN involves two pairs of generators and discriminators learned conjointly, one generator yielding fakes in domain A from latent domain B, and the other generator yielding fakes in domain B from latent domain A:

Contents cannot be rendered at this time; please try again later

(See the Applications section of the NetGANOperator documentation for details on how to train CycleGAN in the Wolfram Language.)

An important detail is that the gradient coming from the discriminator back into the generator is negated, so the generator tries to maximize "LossFake" while the discriminator tries to minimize it. NetGANOperator is pretty equivalent to the following NetGraph, which uses the gradient-negated generator and shares the discriminator at two places (one applied to the real images and one fooled with fake generated images):

In[]:=

ganAsGraph=NetGraph<|"Generator"NetReplacePart[generator,LearningRateMultipliers-1],"Discriminator"->NetInsertSharedArrays[discriminator],"FooledDiscriminator"NetInsertSharedArrays[discriminator],"LossReal"ElementwiseLayer[Function[-Log[#]]],"LossFake"ElementwiseLayer[Function[-Log[1-#]]]|>,



Out[]=

NetGraph



Contents cannot be rendered at this time; please try again later



Intermediate values of this workflow can be computed like this:

In[]:=

ganAsGraph<|"Sample"

,"Latent"NumericArray

Type: Real32

Dimensions: {100}

Data: 0.672, -0.631, 0.975, -0.196, …

|>,NetPort[All]

Out[]=

NetPort[{Generator,Output}]NumericArray

Type: Real32

Dimensions: {3,224,224}

Data: 0.363, 0.48, 0.558, 0.561, 0.576, …

,NetPort[{FooledDiscriminator,Output}]0.0817949,NetPort[{LossFake,Output}]0.0853345,NetPort[{Discriminator,Output}]0.794715,NetPort[{LossReal,Output}]0.229772

What TrainingUpdateSchedule Does Training this in accordance with the typical GAN training loop in the aforementioned pseudo-code can be done by specifying a custom schedule of updates with the new option TrainingUpdateSchedule. Simply using TrainingUpdateSchedule{"Discriminator","Generator"} means that the updates will be done alternatively to the "Discriminator" subnetwork part and to the "Generator" subnetwork part, which is enough for vanilla GAN.

To check this, let’s launch a dummy training with the option TrainingProgressFunction to collect weight values from the discriminator and the generator along the optimization process:

In[]:=

weights={};NetTrainNetGANOperator

,"Latent""Real",

BlockRandom[

]

,All,TrainingUpdateSchedule{"Discriminator","Generator"},TrainingProgressFunctionAppendToweights,

Flatten[

]

&,"Interval"

batch

,



Out[]=

NetTrain Results

summary

batches:20,rounds:10,time:0.014s,examples/s:93091

data

training examples:100,processed examples:1280,skipped examples:0

method

ADAMoptimizer,batch size64,CPU

round

loss:

1.62×

-1

loss real:

6.51×

-2

loss

fake:

9.65×

-2

The progress of the weights shows how they are updated alternatively in the discriminator and the generator. The steps show when weight updates occur:

In[]:=

ListStepPlot[Transpose[weights],PlotLabels{"Discriminator","Generator"},FillingAxis]

Out[]=

It’s also easy to specify that the discriminator should be updated three times, then the generator two times and so on:

In[]:=

weights={};NetTrain

,TrainingUpdateSchedule{"Discriminator"3,"Generator"2},

;ListStepPlotTranspose[weights],



Out[]=

Training GANs to Generate New Pokémon Characters To see how to articulate all these pieces together and learn a generative model with adversarial training, let’s have a bit of fun and try to generate new Pokémon characters. There are 911 images of individual Pokémon characters available in the Wolfram Knowledgebase, which can be obtained efficiently using:

In[]:=

EntityPrefetch[EntityProperty["Pokemon","Image"]];pokemons=DeleteMissing@EntityValue[EntityList["Pokemon"],EntityProperty["Pokemon","Image"]];

We can make these images square by padding as needed, and then resize as 64×64 thumbnails:

In[]:=

pokemons=MapImageResizeImagePad#,

Block[

]

,{64,64}&,pokemons

In[]:=





Inspired by the advances in deep convolutional GANs (DCGANs, as mentioned in Radford et. al, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks,” 2015), our generator will be a chain of

DeconvolutionLayer

interleaved with

BatchNormalizationLayer

and rectified linear units (ReLU):

In[]:=

deconvolutionBlock=Function[{numhiddens,size},NetChain[{DeconvolutionLayer[numhiddens,{5,5},"Stride"{2,2},PaddingSize1],PartLayer[{All,1;;size,1;;size}],BatchNormalizationLayer[],Ramp}]];generator=NetChain[{NetChain[{{512,4,4},BatchNormalizationLayer[],Ramp}],deconvolutionBlock[256,8],deconvolutionBlock[128,16],deconvolutionBlock[64,32],DeconvolutionLayer[3,{5,5},"Stride"{2,2},PaddingSize1,"Weights"0],PartLayer[{All,1;;64,1;;64}],ElementwiseLayer[Tanh[#]*0.5+.5&]},"Input"32,"Output"NetDecoder["Image"]]

Out[]=

NetChain



uniniti

aliz

	Input	vector (size: 32)
1	NetChain (3 nodes)	array (size: 512×4×4)
2	NetChain (4 nodes)	array (size: 256×8×8)
3	NetChain (4 nodes)	array (size: 128×16×16)
4	NetChain (4 nodes)	array (size: 64×32×32)
5	DeconvolutionLayer	array (size: 3×65×65)
6	PartLayer	array (size: 3×64×64)
7	0.5+0.5Tanh[ x ]	array (size: 3×64×64)
	Output	image

FE`NeuralNetworks`Private`NetChain`selected$$1185734240140675668222982455234382420505

FE`NeuralNetworks`Private`NetChain`part$$1185734240140675668222982455234382420505



Our discriminator will be a chain of

ConvolutionLayer

interleaved with BatchNormalizationLayer and leaky ReLU:

In[]:=

convolutionBlock=Function[{numhiddens,size},NetChain[{ConvolutionLayer[numhiddens,{5,5},"Stride"{2,2},PaddingSize2],BatchNormalizationLayer[],ElementwiseLayer[Max[#1,0.2`#1]&]}]];discriminator=NetChain[{ElementwiseLayer[-1+2*#&],convolutionBlock[64,32],convolutionBlock[128,16],convolutionBlock[256,8],convolutionBlock[512,4],LinearLayer[{},"Weights"0,"Biases"None],LogisticSigmoid},"Input"NetEncoder[{"Image",{64,64},"RGB"}]]

Out[]=

NetChain



uniniti

aliz

Input

image

array

(size: 3×64×64)

-1+2

array

(size: 3×64×64)

NetChain (3 nodes)

array

(size: 64×32×32)

NetChain (3 nodes)

array

(size: 128×16×16)

NetChain (3 nodes)

array

(size: 256×8×8)

NetChain (3 nodes)

array

(size: 512×4×4)

LinearLayer

real

LogisticSigmoid

real

Output

real

FE`NeuralNetworks`Private`NetChain`selected$$364927191086715085067839777251166217882

FE`NeuralNetworks`Private`NetChain`part$$364927191086715085067839777251166217882



We can apprehend that training a generator of cartoonish images with less than a thousand images will be challenging. Therefore, we have to start on the right foot and follow all possible good practices. One of them is to first pretrain the generator and the discriminator so that the optimization of GAN will start with good conditions. Let’s try for this to train generative and discriminative subparts in an autoencoder scheme. The goal of an autoencoder is simply to reproduce the input image after compressing it to a vector of 32 values, which will be the latent space of our GAN:

In[]:=

autoencoder=NetChain[<|"compressor"NetTake[discriminator,{All,5}],"latent"NetChain[{LinearLayer[32],BatchNormalizationLayer[]}],"generator"generator|>]

Out[]=

NetChain



uniniti

aliz

Input

image

array

(size: 3×64×64)

compressor

NetChain (5 nodes)

array

(size: 512×4×4)

latent

NetChain (2 nodes)

vector

(size: 32)

generator

NetChain (7 nodes)

array

(size: 3×64×64)

Output

image

FE`NeuralNetworks`Private`NetChain`selected$$9219313556931233775973842482245201765645

FE`NeuralNetworks`Private`NetChain`part$$9219313556931233775973842482245201765645



Autoencoder optimization can consist of minimizing the mean squared error between input and output arrays of pixel RGB values:

In[]:=

autoencoderTrainer=NetGraph[<|"autoencoder"autoencoder,"MSE"MeanSquaredLossLayer[]|>,{{"autoencoder",NetPort["Input"]}"MSE"}]

Out[]=

NetGraph



uniniti

aliz

Contents cannot be rendered at this time; please try again later



An autoencoder to compress images of Pokémon can be trained in a few minutes on a GPU NVIDIA graphics card:

In[]:=

autoEncoderTrainResults=NetTrain[autoencoderTrainer,pokemons,All,TargetDevice"GPU",MaxTrainingRounds300]trained=autoEncoderTrainResults["TrainedNet"];

Out[]=

NetTrain Results

summary

batches:4500,rounds:300,time:3.6min,examples/s:1346

data

training examples:911,processed examples:288000,skipped examples:0

method

ADAMoptimizer,batch size64,GPU

round

loss:

1.4×

-3

	rounds
loss	Contents cannot be rendered at this time; please try again later

The trained autoencoder is able to reproduce the input with a bit of blur:

In[]:=

autoencoder=NetReplacePart[trained[["autoencoder"]],{"Input"NetEncoder[{"Image",{64,64}}],"Output"NetDecoder["Image"]}]

Out[]=

NetChain



Input

image

array

(size: 3×64×64)

compressor

NetChain (5 nodes)

array

(size: 512×4×4)

latent

NetChain (2 nodes)

vector

(size: 32)

generator

NetChain (7 nodes)

array

(size: 3×64×64)

Output

image

FE`NeuralNetworks`Private`NetChain`selected$$8790362301630302558889417573253539900126

FE`NeuralNetworks`Private`NetChain`part$$8790362301630302558889417573253539900126



In[]:=

AssociationMap[autoencoder,RandomSample[pokemons,5]]

Out[]=







At least, we are now sure that the generator architecture is able to generate real Pokémon images out of 32 random values. Pretrained layers can be extracted from this autoencoder to build a generator and a discriminator that are more than ready to be adversarially trained:

In[]:=

generator=NetReplacePart[autoencoder[["generator"]],"Output"NetDecoder["Image"]]

Out[]=

NetChain



	Input	vector (size: 32)
1	NetChain (3 nodes)	array (size: 512×4×4)
2	NetChain (4 nodes)	array (size: 256×8×8)
3	NetChain (4 nodes)	array (size: 128×16×16)
4	NetChain (4 nodes)	array (size: 64×32×32)
5	DeconvolutionLayer	array (size: 3×65×65)
6	PartLayer	array (size: 3×64×64)
7	0.5+0.5Tanh[ x ]	array (size: 3×64×64)
	Output	image

FE`NeuralNetworks`Private`NetChain`selected$$5731396335425211762916503082491668163656

FE`NeuralNetworks`Private`NetChain`part$$5731396335425211762916503082491668163656



In[]:=

discriminator=NetAppend[autoencoder[["compressor"]],LinearLayer[{},"Weights"0,"Biases"None],"Input"NetEncoder[{"Image",{64,64}}]]

Out[]=

NetChain



Input

image

array

(size: 3×64×64)

-1+2

array

(size: 3×64×64)

NetChain (3 nodes)

array

(size: 64×32×32)

NetChain (3 nodes)

array

(size: 128×16×16)

NetChain (3 nodes)

array

(size: 256×8×8)

NetChain (3 nodes)

array

(size: 512×4×4)

LinearLayer

real

Output

real

FE`NeuralNetworks`Private`NetChain`selected$$7644092750498046003306549736515564812103

FE`NeuralNetworks`Private`NetChain`part$$7644092750498046003306549736515564812103



In[]:=

NetGANOperator[{generator,discriminator}]

Out[]=

NetGANOperator



Parameters
Generator :	NetChain (7 nodes)
Discriminator :	NetChain (6 nodes)
Loss :	Wasserstein
Input Ports
Sample :	image
Latent :	vector (size: 32)
Output Ports
LossReal :	real
LossFake :	real
GeneratedFake :	image

FE`NeuralNetworks`Private`FormattingLayers`part$$29797000562671952509912353333820380070



The training of GANs involves being able to infinitely sample random latent values. This can be done by using the generator syntax, as explained in the out-of-core training tutorial, where the third argument of NetTrain is a custom generator function yielding batches of inputs for the training network. For our Pokémon training, the generator function can be written as:

In[]:=

datagen=Function[<|"Sample"RandomSample[pokemons,#BatchSize],"Latent"getRandomLatent[#BatchSize]|>];

... where getRandomLatent is a custom function that samples random numbers from a simple latent distribution—for instance:

In[]:=

getRandomLatent=Function[batchSize,RandomReal[1,{batchSize,32}]];

After some preliminary experiments, we found out that better results can be achieved by normalizing the value in the latent space so that it is contained in the unit sphere. We achieve this using the nonlinear squashing function introduced in Hinton’s capsule networks:

In[]:=

squash=NetGraph

,"Input"{Automatic}

Out[]=

NetGraph



Contents cannot be rendered at this time; please try again later



This squashing function normalizes (latent random) vectors so that their norm never exceeds 1, and preserves the bell shape of the normal distribution, as shown here in two dimensions:

In[]:=

Histogram3D@squash@RandomVariate[NormalDistribution[],{5000,2}]

Out[]=

Contents cannot be rendered at this time; please try again later

We will use this custom latent random distribution:

In[]:=

getRandomLatent=Function[batchSize,squash@RandomVariate[NormalDistribution[],{batchSize,32}]];

For a batch size of 16, the inputs to NetGANOperator for each training update looks like this:

In[]:=

MapAt[ArrayPlot,datagen[<|"BatchSize"16|>],"Latent"]

Out[]=

Sample

,Latent



With two pretrained networks that have sensible architectures and a decent random-number generator, everything is now ready to train the GAN—except maybe the monitoring part. That must be done with a function that shows or records generated examples in order to judge how well the generator performs while training progresses:

In[]:=

monitor=With[{latents=getRandomLatent[25]},Function[generator,ImageCollage[generator[latents],ImagePadding1]]];

At the beginning of the training, generated images look like this:

In[]:=

trainedGan=NetTrain[NetGANOperator[{generator,discriminator}],datagen,TrainingUpdateSchedule{"Discriminator"3,"Generator"3},BatchSize64,MaxTrainingRounds20000,TargetDevice"GPU",TrainingProgressReporting{monitor[NetExtract[#Net,"Generator"]]&,"Interval"

500

rounds

}];

Out[]=

Contents cannot be rendered at this time; please try again later

After a few hours of training on a GPU, the GAN has converged and we can extract the trained generator:

In[]:=

generator=trainedGan[["Generator"]]

Out[]=

NetChain



Input port:	vector (size: 32)
Output port:	image



In the end, here are the new kinds of Pokémon characters that are produced:

In[]:=

newPokemons=generator[getRandomLatent[112]];ImageAssemblePartitionSortBynewPokemons,DominantColors#,

&,8,



Out[]=

Contents cannot be rendered at this time; please try again later

I hope you can see Pokémon here the same way you can see animals in the clouds sometimes. One thousand Pokémon thumbnails were manifestly not enough for a computer to learn all the features that make a Pokémon credible in terms of physiological constitution. But it was enough to learn the texture, the associations of colors and the kinds of curves to make fishy, birdy and doggy shapes in the cartoon world.

This Pokémon trial was not so bad, and there are several possible ways to improve on this first approach and get a higher level of detail. For example, progressive growing of GANs is promising; it would consist of a loop of NetTrain, growing the resolution iteratively by adding layers to the generator and the discriminator. But let’s stop here with Pokémon. The message is that a GAN is not a completely out-of-the-box tool where you put in your data, click a button and just wait. We tried to make it as easy as possible in the Wolfram Language, for example with sensible default option values:

◼

If not specified, the loss function (second argument of NetGANOperator) is automatically set to either "JensenShannon" or "Wasserstein", depending on the output layers of the generator

◼

The default optimization parameters of NetTrain (learning rate and momentum) are automatically tuned to values that are commonly used for GANs, when some NetGANOperator is detected inside the net to train

But still… Many details about the network architecture, the distribution of the latent space and the optimization schedule have to be taken care of depending on the targeted application.

On the Impact of GANs on Society The use of GANs to create deep fakes, where a person is superimposed over an image or a video—possibly in an embarrassing situation—raises techno-ethics questions. The same sort of controversy emerged when transformers (a new kind of neural net architecture) achieved incredible performance in artificial text generation, in turn awaking fears of escalation of fake news on the web.

What about GANs in this situation? It seems that they constitute new, powerful tools to assist and augment human artistic creativity. Their accomplishments arouse the curiosity of everybody. But the “fakeness” of generated results also raises, justifiably, questions about possible misuses. The black box aspect of deep learning also calls for further research on model interpretability of GANs. Obviously, an innovation in machine learning like GANs makes us feel closer to the point at which computer AIs pass the Turing test, bringing with that all the fears of catastrophic scenarios that could happen in case humans lose control.

It’s not new that a powerful technology can be misused by malicious people. And it’s important that people in the research community are aware of the possible deviations of the new, powerful technology they publish. But in the end, all the actors related to the use of these technologies (not only researchers, who are just the early actors) must be well intentioned and make efforts to ensure that automatic algorithms don’t get out of control. What’s particularly sensitive with artificial generative models is that being benevolent is not enough to avoid a negative impact. The creators of the models must benchmark correctly the performances of the published models and models in production to avoid problems like unintended social biases (a model that discriminates by characteristics such as race, age, gender, etc.). But they cannot be responsible for the bad intentions of others. So the role of the researchers is to educate users of their publications toward the mastery of standard techniques to control social biases and ease model interpretability in general.

Try It Yourself Future developments on NetGANOperator will include extensions to multiclass classification and to generative models with multiple inputs. Post your best GAN experiments in the comments or share them on Wolfram Community. Questions or suggestions for additional functionality are also welcome.

Cite this as: Jérôme Louradour, "Generative Adversarial Networks (GANs) in the Wolfram Language" from the Notebook Archive (2020), https://notebookarchive.org/2020-08-8blivpl