Is there any possibility to apply deep dreaming in data augmentation?

Question

I looked into the deep dreaming concepts and feel like this has the potential for data generation. But i'm not sure how possible this concept is. Any thoughts regarding this?

Answer 1

TL;DR Not usually, because Deep Dream ignores the distribution of your input population.

You cannot extract additional informative data using generators, only remix and recombine what you already have. There may be a few situations where that is useful because you can use a generator trained on other data to bring in extra relevant information that was not in your main dataset.

---

Deep Dream logically alters an input structure to maximise some measure within a neural network based on neuron activations. This might be the activation of a single neuron, or of a layer of neurons.

I have worked with Deep Dream scripts based on Google's Inception v5 network which was trained on ImageNet, just for fun.

Here's a relatively extreme example of output, with a few specific neuron activatioons maximised and a long zoom so that the data is essentially only a super-stimulous for those artifical neurons:

One thing you can say with reasonable certainty is that this image does not look like anything you might draw at random from ImageNet, from any category. It would be a poor choice as augmentation for ImageNet classification tasks, because it is not from the target population.

The same is true in general for all obvious Deep Dream modifications. The modifications do not look natural and produce inputs that would not reasonably be in the input distribution. Once you are clearly outside the input distribution for your use case, then learning how to process those inputs may not give you any benefit related to the goals of your machine learning task.

More subtle changes might be useful, if on inspection it is clear that they should not change the categorisation. This might be similar augmentation to adding noise, and if the alterations were chosen carefully, might help defend against adversarial inputs. In fact generating adversarial inputs is an almost identical process to Deep Dream, usually focusing on the output categories and updates that look like high frequency noise so that humans tend not to see the difference.

There are better data generating models that attempt to create data that could have come from the input population. Two major ones are variational autoencoders (VAE) and generative adversarial networks (GAN). These might be useful for data generation.

However, if you are considering somehow "bootstrapping" a small dataset by training one of these generators on it then using its output to augment your data, it is unlikely that this will work. The quality of the output will usually depend on its training data such that generated data will not truly add information that you did not already have in the original training set. In most cases you will better off looking at simpler logical transformations of data that you are confident will not change the target label (or that change it in predictable ways) - for image-based CNNs that might include changing rotation, crop, brightness/contrast and other basic image manipulations.

There might be a few cases in which working with a generator helps. For instance if you can train it on far more data because you have lots of unlabelled data, or dat afrom a different but similar task. Then the quality of generated inputs could be high enough that you effectively get to make use of unsupervised learning from the same dataset. You might still find it easier to do transfer learning, or train an autoencoder and re-use its first few layers though.