Using neural network for automatic software performance tuning

Question

I am currently testing performance of two different messaging systems and there are many independent tuning and load parameters that influence the message rate. I would like to develop an automated method that would produce the maximum message per second rate on a given hardware.

CONSTANTS:

• # of CPUs, memory, network and other hardware parameters are a constant (or at least there is a known maximum).
• Version of OS and the messaging products is a constant (WebSphere MQ 7.5 for instance)
• Message size
• 3 minutes to complete one test measurement

DESIRED RESULT:

• Maximum rate of messages per second

VARIABLES:

• OS tuning options (kernel, TCP and other network and memory stuff)
• Messaging software tuning settings - of which there are several dozens (buffer sizes of all sorts, number of threads, number of Queue Managers, number of queues, different kinds of bindings for clients, security settings, log types, number and sizes of log and data files, etc.....)
• Number of message producers and consumers (say a range from 1 to 10,000)

Currently performance tuning is done as an iterative process - change one variable at a time, measure msg/sec, rinse and repeat.

I have developed a script framework that can iterate over different settings, but if I do this as a nested loop that is 50 levels deep and iterate over every possible value for each variable this process will take hundreds of years to complete. Including a random factor might help, but what I would like to do is to use neural network to find optimum settings for my tuning variables.

I found some research papers that talk about this kind of thing, but could not find any real implementations. Has anybody done it? Any recommendations on how to proceed? I have not worked with neural networks, but have done mathematical programming (optimization) in university and normal optimization methods will take too long to solve this problem.

Answer 1

This kind of problem doesn't sound very suitable for neural networks. Neural networks are probably most often used as classifiers with supervised learning. Typically, you have to have a training set with examples that are already classified. There are some unsupervised approaches too, most recently most notably perhaps the deep neural networks. But that is an active area of research and I doubt there are ready made tools for what you need.

You have a problem with combinatorial explosion in the parameter space. That kind of problem is often well suited for solving with evolutionary algorithms. For example genetic algorithms or simulated annealing.

With genetic algorithms the problem solving shifts to finding

• a good problem representation that allows you to do crossover and mutations,
• a suitable fitness function.

This often proves to be very difficult.

Your problem, however, seems to be very suitable for this method:

• individual solution representation: a list of parameters i = [p_1, p_2, ..., p_n]
• crossover of i_1 and i_2: choose a splitting point between 1 and n and exchange the respective parts of the individuals' lists
• mutation: choose a parameter and adjust it
• fitness function: rate of messages per second

It's not guaranteed that you would get an optimal solution this way but it can help you to combat the combinatorial explosion quite effectively.