Neural network with single positive non limited output value

Question

Can somebody help me with selecting right activation function in neural network. The netwrork is written in Python and it should approximate real estate price(output) based on its characteristics (inputs).

Now I'm using sigmoid activation function on both, output and hidden layers. In all train cases price was normalized ( item.price / max_price_of_all_items) so on the output I have values < 1. Here some more characteristics I'm using:

• learning speed is 0.01
• training set about contains ~600 entries (can be up to 3500)
• Hidden layers 2
• Neurons in layer 5
• inputs 9
• output 1
• epochs - 5000 (not enough?)

Trained network gives very poor results. Is there something I've missed. Is it possible to have normal price on the output (not values from [0:1) like now). Should I change activation function on output neurons.

Thanks for any help

Answer 1

Here are a few suggestions:

1. Make sure you are using a mean squared error on the output, no activation function needed: err(x,f(x))= (x-f(x))^2, where x is the true output and f(x) is the predicted output.

2. If you implemented back propagation yourself, track the progress of the average error of the network for each epoch. If the error increases between epochs, then this indicates that your implementation is incorrect. You could also check the gradients computed by backpropagation by comparing with approximate gradients computed using finite differences.

3. Normalize your input and output: subtract the means and divide by the standard deviation or whiten your data using PCA

Resources:

• http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial
• http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf

Answer 2

You are asking a very vague question on a very broad topic but relating to your very concrete and individual problem. You'll probably not get a helpful answer this way but I'll give it a try anyway.

By your description it seems like you are new to the field of neural networks and would like to just throw it on your classification problem. As you might infer from the length and depth of the wiki page on neural networks there are lots of dificulties involved in making neural networks "just work". I recommend studying the topic a bit before expecting insightful results. You'll have a much better idea where to look for what's wrong with your approach.

Here is my list of things you should pay attention to when using neural networks for classification:

1. Data sets for training and validation
2. Input encoding and normalization
3. Neuron types (activation function)
4. Network topology
5. Output encoding and "denormalization"
6. "Fitness" evaluation function
7. Training algorithm

My best guess is that you got a good amount of those right already but your topology is insufficient. Topology is what makes or breaks the ability to learn stuff! You might want to add more neurons in the hidden layer. Also you might want to use multiple output neurons instead of one. In your case you might have real estate value categories ($0-10k, $10k-50k, $50k-100k, etc.) so you could use one output neuron per category. Those categories might be easier to learn than a precise analog estimate.

My second best guess is that there simply is no pattern in your input data. If you were presented one input vector, would you be able to estimate the correct real estate value? How do you do it?

If none of my guesses are helpful, it seems like something with your training process is not working correctly. Investigate by applying your implementation to popular and well understood training problems on the internet to gain more insight.