Prototyping neural networks
https://stackoverflow.com/questions/1846592
-
12-09-2019 - |
italiano
english
français
española
中国
日本の
العربية
Deutsch
한국어
Português
RussianQuestion
from your experience, which is the most effective approach to implement artificial neural networks prototypes? It is a lot of hype about R (free, but I didn't work with it) or Matlab (not free), another possible choice is to use a language like C++/Java/C#. The question is mainly targeting the people that tried to test some neural networks architectures or learning algorithms.
If your choice is to use a programming language different from the three mentioned above, can you tell me their names and some explanations concerning your choice (excepting: this is the only/most used language known by me).
Thanks.
Solution
Since neural networks are used a lot by scientists, and not so much by programmers, choose a programming language that targets scientists and provides good libraries for neural networks.
Python is a reasonable choice since it is widely used by scientists. Two distributions to start with are
These Python distributions include a lot of add-on modules that are not in the standard Python library but which are very useful for the type of programming that scientists do. Searching the Python package index comes up with a few neural network packages that may be OK if you are just learning about neural networks.
http://pypi.python.org/pypi?:action=search&term=neural&submit=search
However, if you are doing serious work with Neural Networks you will want something like the Fast Neural Network library. This comes with Python bindings so that you can do your programming in Python, use the wide variety of Python modules for graphing, visualisation, data manipulation and so on. But, your neural networks will run using optmised compiled code from the FANN library. The best of both worlds.
In other words, for running the actual neural network code, you need C, not Java. Since C libraries don't integrate that well with Java, choose a language that does integrate smoothly with C libraries. Python does that and is also rather more productive than Java since there is a lot less lines of code required to explain your algorithms. Some people have found a 10 times increase in productivity over Java.
You mentioned R, perhaps because it has statistical functions that you would need to use, or perhaps you have people available who can write R code. Again, choosing Python versus R is not an either-or decision. You can use both.
The RPY library allows Python programs to access R libraries and code. Using this you would write your main programs in Python and treat R as a tool to provide libraries, in the same way that you make use of the FANN library which was written in C.
There is another module called RSPython that works both ways, so that R programs can access libraries written in Python. This would be useful if you have an expert R programmer helping you.
http://www.omegahat.org/RSPython/
And that's not all. You can leverage Python to simplify Java programming. If you have a Java neural network engine, you can still write most of your program in Python using the Jython version which runs on the Java VM and allows you to use any Java libraries and classes in your code. And you can still use the fast FANN library as well since they provide bindings for Java programs.
The main reason that I recommend Python for your work is that it is used by an awful lot of scientists which is why there are two science oriented distributions available. A second reason is that Python is very easy for novice programmers to get started with, and in exploring neural networks you will probably start with simpler simulations and work up to more complex ones with more data manipulation and analysis of results. Python will allow you to build up your own library of code, and become an expert Python programmer so that you can focus more of your attention on the neural network problems. If you bounce around with a bit of Java, and a bit of C++ and a bit of R, then you will be able to spend less time on neural networks. That strategy might be good for someone who wants a career in programming, but not for someone who wants to achieve some significant results with neural networks.
Even if your neural network work extends into specialised hardware, so-called neuromorphic chips, you can still leverage Python as this paper from the NIH demonstrates:
OTHER TIPS
I tried using both higher-abstraction level languages (matlab, java) and lower ones (C). Both using toolboxes and libraries and coding them myself. The overall response? Neither is the perfect tool. Please keep in mind that:
a prototype might not be enough: often you need to run the network over large samples, or several times over a subset of samples (in case of evolving neural networks) to get decent results. If you have to run a network a million times, even a small performance gain might be a huge help and time-saver (i.e. C over matlab);
if, on the other hand, you need ease of coding you might want to use one of the many pre-packaged libraries (javaNN, etc);
what kind of neural network are you using? continuous-time recurring neural networks (CTRNN)? Backprop? How do you do train them? How do you verify their results? Is accuracy important? (i.e. are you running them on a small device, such as a memory-limited robotic control board, like Arduinos?)
If you have time to spare, I'd suggest
- learn the concepts using a higher-level language, or even pseudocode first;
- once you are familiar with all intricacies, especially if using evolving neural networks, choose a language they are already familiar with
- then you might want to start researching how to optimize for speed, memory footprint, etc.
Hope this helps.
I have used Matlab's Neural networking tool box to some extent, and enjoyed it as a prototyping tool due to its graphic interface to allow you to set up the network, experiment with sizes of training vs test input data, and its post-training testing all built in. It just seemed nice and natural to get started with...
I also experimented with Joone (Java Object Oriented Neural Engine). It is set up quite nicely, and at the time I was a bit of an amateur and still had no problems getting various networks up and running, with testing. It is Java, so it may not have the performance that you might want if you are training large complex systems, but its API was extremely approachable.
I have also seen some work done with Flood in C++. It has a bunch of classes set up with networks that are appropriate to solve a range of problems. It is worth checking out.
I would also highly recommend python. For a prototype, python is a great choice: it is easier and faster to program in, there are a huge number of libraries available, and it widely used in the scientific community.
Also, if you are using python, you can also take advantage of the excellent PyBrain package which contains algorithms for neural networks, reinforcement learning, unsupervised learning, and other machine learning tasks, which should help you build a prototype quickly.
This depends on your current setup. When I used to work on them back in college days, I had to use C++ + MPI+numerical recipes. This was done because I had to load share on big beowulf cluster.
If your computation needs are not big, anything would do. Prepackaged libraries are available on all the platforms (R, Python(numPy, scipy), C/C++(Numerical recipes) etc). If you are comfortable programming in any one of them, it should not be a big deal.
If I had to prototype anything now, I'd probably go with Python (just because I find it much more easier for prototyping)
Scilab is an open source alternative to Matlab.
You may experiment with neural networks using the ANN Toolbox for Scilab.
Focus on the theory and experiments. Did you choose discriminant features for your samples? What's the state of your training and test sets. For each experiment, study the confusion matrix. Do you have an idea why a sample is misclassified? Does it seem logical to you? If not, which feature you don't use would help?
Implementing comes next, use the language you're familiar with. A managed language like Java or C# is likely to be less error prone: at least you're less likely to screw up things because of pointer or memory allocation bugs. Optimizing comes last (after some decent profiling as always).
"Encog is an advanced neural network and machine learning framework. Encog contains classes to create a wide variety of networks, as well as support classes to normalize and process data for these neural networks. Encog trains using multithreaded resilient propagation. Encog can also make use of a GPU to further speed processing time. A GUI based workbench is also provided to help model and train neural networks. Encog has been in active development since 2008."
Encog is available for Java, C#.Net and Silverlight.
In my nn classes in school we used matlab and then I used java for my thesis.
I would suggest java or matlab. Matlab because it probably already has alot of what you might need. And java because it is quick to implement what might be missing in open source projects. And besided since besides implementing the neural networks you might need some way to visualize them. And for that I personally believe that java or matlab is quite easy.
I like flood. It's free, comprehensive, and written in C++.
The implementations in Matlab are sophisticated and complete. I have found it to be sufficient for evaluating different types of networks. It is also very programmable using external interfaces.
However, since the implementations of the algorithms are not open source, it is sometimes more difficult when you need to move a particular piece of code into an application outside of Matlab, as my hand coded implementations of different neural network types produced different results.
I have started writing a NN implementation using C++ and found that I didn't know enough about the maths involved in the beginning. What ended up hapenning was that it was too hard to refactor the code as I was tweaking the calculation model.
Eventually I gave into MATLAB as it was definitely a better companion to learning how neural networks work. I was able to make huge changes to the algorithm using a few key strokes and graph the results too.
Perhaps my experience would have been better if I had used an already built matrix computation framework. Considering that's how you do 3D there should be a few really optimized libraries out there for most languages. Heck you might as well leverage Direct3D or OpenGL for that, but I am sure there is something more suitable for Neural Networks out there.
I guess I am not providing much information on what you should do. I can however tell what you should not do - and that's trying to write matrix manipulation code yourself.
You might want to give Weka a look. It has some built-in tools for things like data visualization, and it's been around for years now (some screenshots).
In my view in working with neural networks the key is getting the training set right not so much how the network itself is actualised in code. I would choose a language based on the type of problem you are trying to solve using the network. For the network itself c++, c#, python and java are all viable.
Are you using this in connection with a problem requiring image processing? In which case you will probably want something that hooks up to an image processing library like OpenCV easily. Or is there some audio processing involved?
You may need to easily visualise the training sets so how easy would this be with the language of choice? Can you work with either OpenGL or DirectX libraries either directly or using a wrapper of some kind? For DirectX the choices are C++ and C#. Will a higher level of abstraction say using WPF work?
I have used C# because I am familiar with it and can leverage the numerous data handling technologies in .net and use wpf for any visualisations needed.
You might also want to give Nen a try - It's free, easy to use and very lightweight. It also outperforms LIBSVM in an out-of-the-box-comparison on many popular Regression- and Classification-Datasets.
Typically when I was tinkering with these kinds of algorithms, I found that using the open-source Weka Toolkit was a great way to prototype and discover a whole bunch of different learning algorithms (not just artificial neural networks). These days, it looks as though they have bindings from a bunch of different languages, so you shouldn't be tied to Java if you want to interact with Weka at the code level.
Once I found and understood something that was cool/doing a pretty good classification job, I wrote the algorithm in C or C++ partly for fun and partly to gain the necessary speed gains required to work with larger data sets.
Use C++ and if your neural network is simple enough don't use weird frameworks out there.
Maybe 5 years ago I did a Sokoban solver using Reinforcement Learning. At the time we decided to go with Java and use some Agent Framework developed by an Italian University.
First bad decision here was to use this Framework. It basically had bugs here and there which caused us to waste loads of time debugging the framework's code.
Once we got to the point where everything was stable, training the Neural Network was just dead slow. We left it running overnight in a quite powerful machine and still it solved very few puzzles. Basically Java object allocation and garbage collection were damaging the whole performance of the system quite badly. We tweaked a bit the app by creating pools of objects instead of allocating them all the time but the program's performance was still an order of magnitude less than a similar assignment that was implemented in C++ using sockets directly.
Hope this helps!
Well, if you like to prototype fast then python or matlab seems the best. There are a lot of libraries available for neural networks. Some of the libraries are designed in such a way that they give you minimal amount of tweaking scope but libraries like Theano are extremely fast as the function are internally compiled in C(i guess the fastest till date) and it gives you full functionality of internal tweaking. Theano is typical to use at first but is extremely powerful. And later if you wish to move from neural to deep neural then there is not much of a difference. Professional Kagglers also tend to use this library. This library also has GPU support. Of all the libraries i have used i found theano to be the most useful and powerful. Some of the other state of the art framework for deep learning like Caffe are developed on theano.
Hope this helps.
I've implemented several neural network prototypes in Java and C/C++. When performance matters, use with C/C++.
One of my implementation (not vectorized) of a basic Multilayer Perceptron does a million training iterations (with a setup of 2 input, 4 hidden and 1 output node) in less than a minute. My almost identical implementation in Java takes a huge amout of time to do this. You could also use a linear algebra library like Eigen to create a vectorized version to improve the processing speed.
Imagine you would use yours for any kind of image processing (Face-Recognition, OCR, etc.) with, for example, 28x28 pixel images. You would have 784 input units and at least as many hidden units. This takes a huge amount of time to train so it's beneficial to save precious minutes, hours or days.
