Biowulf High Performance Computing at the NIH
BioGANs: GANs for Biological Image Synthesis

BioGANs is a novel application of Generative Adversarial Networks (GANs) to the synthesis of fluorescence microscopy images of living cells. It allows to infer and visualize the correlated localization patterns of different fluorescent proteins. These patterns reflect important biological functions, but cannot be visualized experimentally for multiple proteins at the same stage of cell growth cycle.


Important Notes

Interactive job
Interactive jobs should be used for debugging, graphics, or applications that cannot be run as batch jobs.

Allocate an interactive session and run the program. Sample session:

[user@biowulf]$ sinteractive --mem=40g --gres=gpu:p100,lscratch:10 -c8
[user@cn3200 ~]$module load biogans  
[+] Loading cuDNN 7.0  libraries...
[+] Loading CUDA Toolkit  9.0.176  ...
[+] Loading gcc  7.2.0  ...
[+] Loading libtiff libraries v 4.0.9  ...
[+] Loading biogans  20191230
The BioGANs application, as it is implemented on Biowulf, comprises the following three executables:
- takes as input two-channel fluorescence microscopy images from the Localization Interdeoendency Network (LIN) dataset
  and performs training of any of the three available neural network architrectures,
    (1) DCGAN (=default),
    (2) DCGAN-separable, or
    (3) DCGAN-starshaped,
   using any of the three available GAN algorithms:
    (1) (traditional) GAN (=default),
    (2) Wasserstein GAN (WGAN), or
    (3) WGAN with gradient penalty (WGAN-GP).
- takes as input a trained model in the HDF5 format, generates one "red" and up to six corresponding "green" channels/images that represent localization of different proteins in a cell, and stores the results as HDF5 stack. - takes as input either HDF5 stack or an individual image file and visualizes the data stored in the input file.

To see the entire list of source files and copy it to your current folder, type:
[user@cn3200]$ ls $BIOGANS_SRC  
[user@cn3200]$ cp $BIOGANS_SRC/* . 
To list the executable files, type:
[user@cn3200]$ ls $BIOGANS_BIN
To copy sample data and checkpoints to your current directory, enter the command:
[user@cn3200]$ cp -r $BIOGANS_DATA/* .
This command will create the subfolders data and checkpoints, as well as an empty folder images in your current directory.
[user@cn3200]$ tree -l data
├ LIN_Normalized_WT_size-48-80_test -> /fdb/BioGANs/data/LIN_Normalized_WT_size-48-80_test
│   ├ Alp14
│   │   ├ cell1006.png
│   │   ├ cell10118.png
│   │   ...
│   ├ Arp3
│   │   ├ cell10056.png
│   │   ├ cell10177.png
│   │   ...
│   ├ Cki2
│   │   ├ cell1005.png
│   │   ├ cell10114.png
│   │   ...
│   ├ Mkh1
│   │   ├ cell10075.png
│   │   ├ cell10106.png
│   │   ...
│   ├ Sid2
│   │   ├ cell10009.png
│   │   ├ cell10098.png
│   │   ...
│   └ Tea1
│       ├ cell10058.png
│       ├ cell10071.png
│       ...
└ LIN_Normalized_WT_size-48-80_train -> /fdb/BioGANs/data/LIN_Normalized_WT_size-48-80_train
    ├ Alp14
    │   ├ cell100005.png
    │   ├ cell100087.png
    │   ...
    ├ Arp3
    │   ├ cell100057.png
    │   ├ cell100079.png
    │   ...
    ├ Cki2
    │   ├ cell100026.png
    │   ├ cell100044.png
    │   ...
    ├ Mkh1
    │   ├ cell100015.png
    │   ├ cell100242.png
    │   ...
    ├ Sid2
    │   ├ cell100002.png
    │   ├ cell100014.png
    │   ...
    └ Tea1
        ├ cell100077.png
        ├ cell100081.png
The BioGANs implementation on Biowulf includes only a part of the entire LIN dataset, comprising 22,876 training and 4,033 testing images.
To list available command line options for the script biogans_train.pyR, type:
[user@cn3200]$ -h 
Using TensorFlow backend.
usage: [-h] [-a GAN_ALGORITHM] [-b BATCH_SIZE] -d DATAROOT
                        [--epochs EPOCHS] [-g num_gpus] [--lrD LRD]
                        [--lrG LRG] [-m NETWORK_MODEL] [-M MONITOR_STEP]
                        [--num_disc_iters NUM_DISC_ITERS] [--ngf NGF]
                        [--ndf NDF] [-p checkpoint_prefix]
                        [--random_seed RANDOM_SEED] [-w] [-z NZ]
                        [--wgan_clip_value WGAN_CLIP_VALUE]
                        [--wgangp_lambda WGANGP_LAMBDA] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -a GAN_ALGORITHM, --gan_algorithm GAN_ALGORITHM
                        GAN algorithm to train: GAN | WGAN | WGAN-GP
  -b BATCH_SIZE, --batch_size BATCH_SIZE
                        input batch size; default=64
  --epochs EPOCHS, -e EPOCHS
                        number of iterations to train for
  -g num_gpus, --num_gpus num_gpus
                        number of gpus to use; default=1
  --lrD LRD             learning rate for discriminator/critic, default:
                        depends on gan_algorithm and optimizer
  --lrG LRG             learning rate for Generator, default: depends on
                        gan_algorithm and optimizer
  -m NETWORK_MODEL, --network_model NETWORK_MODEL
                        Architecture of the model: DCGAN | DCGAN-separable |
  -M MONITOR_STEP, --monitor_progress MONITOR_STEP
                        visualize generated images after each monitor_step
  --num_disc_iters NUM_DISC_ITERS
                        Number of iterations of the WGAN critic per one update
                        of generator
  --ngf NGF             number of generator filters
  --ndf NDF             number of discriminator filters
  -p checkpoint_prefix, --checkpoint_prefix checkpoint_prefix
                        prefix of the output checkpoint file .*.h5
  --random_seed RANDOM_SEED
                        Random seed, default - the answer to the ultimate
  -w, --use_pretrained_weights
                        load pre-trained model weights
  -z NZ, --nz NZ        size of the latent z vector
  --wgan_clip_value WGAN_CLIP_VALUE
                        for WGAN
  --wgangp_lambda WGANGP_LAMBDA
                        for WGAN-GP
  -v, --verbose         increase the verbosity level of output

required arguments:
                        Path to the training dataset

Here are examples of the training commands:
[user@cn3200]$ -d data/LIN_Normalized_WT_size-48-80_train/Alp14 -m DCGAN -M 10 -a GAN  
[user@cn3200]$ -d data/LIN_Normalized_WT_size-48-80_train/Alp14 -m DCGAN-separable -M 10 -a WGAN  
[user@cn3200]$ -d data/LIN_Normalized_WT_size-48-80_train -e 2000 -a WGAN-GP -m DCGAN-starshaped 
Using TensorFlow backend.

Data loading: dataset_type == folder

100%|█████████████████████████████████████████████████████████████████| 3262/3262 [00:13<i;00:00, 246.98it/s]
100%|█████████████████████████████████████████████████████████████████| 3566/3566 [00:14<i;00:00, 246.12it/s]
100%|█████████████████████████████████████████████████████████████████| 4745/4745 [00:19<i;00:00, 249.28it/s]
100%|█████████████████████████████████████████████████████████████████| 2614/2614 [00:10<i;00:00, 249.52it/s]
100%|█████████████████████████████████████████████████████████████████| 4151/4151 [00:16<i;00:00, 247.98it/s]
100%|█████████████████████████████████████████████████████████████████| 4538/4538 [00:18<i;00:00, 248.15it/s]

Data normalization...

Compiling models...

Total params: 16,594,950
Trainable params: 16,584,198
Non-trainable params: 10,752
failed to get the current screen resources
[epoch 10/10000] D_loss:   0.5292916205033291; G_loss:   1.2565401772658029; Ftime: 0.169202h
[epoch 20/10000] D_loss:   0.2152213265927096; G_loss:   1.2955409357945125; Ftime: 0.302744h
[epoch 30/10000] D_loss:   0.2142432650327765; G_loss:   1.0367570132017134; Ftime: 0.436208h
[epoch 40/10000] D_loss:   0.2260407583397055; G_loss:   0.8711004773775736; Ftime: 0.569363h
[epoch 50/10000] D_loss:   0.2143620753713888; G_loss:   0.7085781350731850; Ftime: 0.702427h
[epoch 60/10000] D_loss:   0.2237661391874523; G_loss:   0.5985602905352910; Ftime: 0.836148h
[epoch 70/10000] D_loss:   0.2142693935566109; G_loss:   0.6081153067449729; Ftime: 0.970462h
[epoch 80/10000] D_loss:   0.2145741402819311; G_loss:   0.4473370803520084; Ftime: 1.103888h
[epoch 90/10000] D_loss:   0.2142372822959699; G_loss:   0.3972657910897396; Ftime: 1.237428h
[epoch 100/10000] D_loss:   0.2142354197664384; G_loss:   0.1745595527764332; Ftime: 1.371228h
[epoch 110/10000] D_loss:   0.2182818037449373; G_loss:   0.2771052525349660; Ftime: 1.504606h
[epoch 120/10000] D_loss:   0.2142665025198647; G_loss:   0.3534661421746326; Ftime: 1.638297h
Note that running of the latter command takes very long, it should normally be submitted as a batch job.

During the training procedure, the following output foolders will be created (if they did not exist previously) in your current working directory:
checkpoints   images
Stored in the first of these folders will be the following files in HDF5 format:
(1) weights of the generator, discriminatori/critic and combined models,
weights.<base training folder>.<model type>.<network_architecture>.<gan_algorithm>.h5
which can be used by the executable to continue the training procedure from the stored checkpiont.
(2) generator model(s), which can be used as input by the executable
model.generator.<base training folder>.<model type>.<network_architecture>.<gan_algorithm>.h5
For example:
[user@cn3200]$ ls checkpoints      
With command line option -M passed to the executable, the prograss of the training procedure can be monitored by storing samples of generated images in the file
These samples can be visualized using the executable
[user@cn3200]$ -h
usage: [-h] -i input_file [-v]

optional arguments:
  -h, --help     show this help message and exit
  -v             output debugging info

required arguments:
  -i input_file  The file to be visualized
For example:
[user@cn3200]$ -i images/monitor.GAN.DCGAN-separable.png

[user@cn3200]$ -i images/images/stack7.DCGAN-starshaped.GAN.h5
The executable supports the following command line options:
[user@cn3200]$ -h
Using TensorFlow backend.

usage: [-h] [-e] [-d DATAROOT] [-D] -i input_file
                          [--optimizer OPTIMIZER] [--random_seed RANDOM_SEED]
                          [-v] [-z NZ]

optional arguments:
  -h, --help            show this help message and exit
  -e, --evaluate        evaluate predicted images by comparing with real data
                        Path to the training dataset
  -D, --debug           output debugging info
  --optimizer OPTIMIZER
                        Optimizer to use for training: default (depends on
                        gan_algorithm) | adam | rmsprop
  --random_seed RANDOM_SEED
                        Random seed, default - the answer to the ultimate
  -v, --verbose         increase the verbosity level of output
  -z NZ, --nz NZ        size of the latent z vector

required arguments:
  -i input_file, --input_file input_file
                        Saved generator model or weights file
To run this executable on a saved generator model file, type
[user@cn3200]$ -i checkpoints/model.generator.Alp14.DCGAN-separable.GAN.1.h5 
Using TensorFlow backend.
Output image stack to file: images/stack2.DCGAN-separable.GAN.h5             
Now visualize the predicted images:
[user@cn3200]$ -i images/stack7.DCGAN-starshaped.WGAN-GP.h5

End the interactive session:
[user@cn3200 ~]$ exit
salloc.exe: Relinquishing job allocation 46116226
[user@biowulf ~]$