Can evolutionary computation be a method of reinforcement learning?

Question

What is evolutionary computation? Is it a method of reinforcement learning? Or a separate method of machine learning? Or maybe none?

Please, cite references used to answer this question.

Answer 1

There are evolutionary methods that are explicitly aimed at solving the reinforcement learning problem. The subfield typically goes by the name of Learning Classifier Systems (LCS) or occasionally Genetics-Based Machine Learning (GBML).

Aside from that, I'm not sure your question has a very well-defined answer. It basically boils down to "what is machine learning?" There's no canon that we've all agreed on for how to answer that question. For some, EC might be a part of that subfield. For others, it isn't. I just sampled a handful of ML textbooks from my shelf, and about half contained material on evolutionary methods. I suspect 15 years ago that fraction would have been higher, but fashions change, and machine learning is very nearly a subfield of statistics now. EC methods don't fit that mold very well.

Answer 2

Evolutionary computation, or evolutionary algorithms, are optimization algorithms, which, when applied to a neural network (as in neuro-evolution) can certainly be classified as a form of reinforcement learning, although it works a bit different than the usual reinforcement learning algorithm.

Generally, in evolutionary algorithms such as genetic algorithms, or evolution strategy, you have a whole population of individuals to be optimized. For each of those individuals, a quality function is used to determine their 'fitness' (as in 'survival of the fittest'), and the best individuals are selected for the next generation. Those 'parents' are then randomly duplicated, modified, mutated, or even recombined with each others -- how exactly this is done is a bit different in each of the different algorithms. Finally, those new mutated and/or recombined parents form the population for the next generation, and the process starts again, until some desired quality is reached, or the quality levels out.

In the case of neuro-evolution, the individuals are neural networks, which are mutated by randomly changing weights (whereas in classical neural networks the weights are updated according to very precise mathematical rules) or even altering their topology, and the quality of the individuals is determined by how well they perform on the training data.

Sorry, no hard scientific reference here, but maybe this still helped clearing things up a bit.

Answer 3

The principle difference between reinforcement learning [1] and evolutionary computation [2] is that RL in the original sense is applied to an agent in an environment, learning a policy (also see the Wikipedia article on reinforcement learning), while EC is a more generic term for a class of search algorithms that use 'evolutionary' inspired methods for optimizing the search. I wouldn't classify EC as machine learning at all, and I haven't found a source that does.

[1] Reinforcement learning: An introduction - RS Sutton, AG Barto - 1998 - Cambridge Univ Press

[2] What is evolutionary computation? - DB Fogel - Spectrum, IEEE, 2000

Answer 4

Update as of 2017: The answer is YES. The most downloaded paper over the past month in Reinforcement Learning, aptly named "Evolution Strategies as a Scalable Alternative to Reinforcement Learning" is indeed the talk of the town.

Answer 5

So, where is evolutionary computation? Is it a method of reinforcement learning? Or a separate method of machine learning? or maybe none?

I view EC and ML as distinct from each other. However, there are some great applications out there whey they are used in conjunction. Although this has been a small area of research for quite some time, I feel there is some low-hanging fruit in the area of combining EC and ML. I think a lot of people don't have the patience to see those ideas through.