What is the difference between a Bayesian network and a naive Bayes classifier?

Question

What is the difference between a Bayesian network and a Naive Bayes classifier? I noticed one is just implemented in Matlab as classify the other has an entire net toolbox.

If you could explain in your answer which one is more likely to provide a better accuracy as well I would be grateful (not a pre-requisite).

Answer 1

Short answer, if you're only interested in solving a prediction task: use Naive Bayes.

A Bayesian network (has a good wikipedia page) models relationships between features in a very general way. If you know what these relationships are, or have enough data to derive them, then it may be appropriate to use a Bayesian network.

A Naive Bayes classifier is a simple model that describes particular class of Bayesian network - where all of the features are class-conditionally independent. Because of this, there are certain problems that Naive Bayes cannot solve (example below). However, its simplicity also makes it easier to apply, and it requires less data to get a good result in many cases.

Example: XOR

You have a learning problem with binary features x1 and x2 and a target variable y = x1 XOR x2.

In a Naive Bayes classifier, x1 and x2 must be treated independently - so you would compute things like "The probability that y = 1 given that x1 = 1" - hopefully you can see that this isn't helpful, because x1 = 1 doesn't make y = 1 any more or less likely. Since a Bayesian network does not assume independence, it would be able to solve such a problem.

Answer 2

Naive Bayes is just a restricted/constrained form of a general Bayesian network where you enforce the constraint that the class node should have no parents and that the nodes corresponding to the attribute variables should have no edges between them. As such, there is nothing that prevents a general Bayesian network from being used for classification - the predicted class is the one with the maximum probability when (conditioned on) all the other variables are set to the prediction instance values in the usual Bayesian inference fashion. A good paper to read on this is "Bayesian Network Classifiers, Machine Learning, 29, 131–163 (1997)". Of particular interest is section 3. Though Naive Bayes is a constrained form of a more general Bayesian network, this paper also talks about why Naive Bayes can and does outperform a general Bayesian network in classification tasks.

Answer 3

For the Bayesian network as a classifier, the features are selected based on some scoring functions like Bayesian scoring function and minimal description length(the two are equivalent in theory to each other given that there are enough training data). The scoring functions mainly restrict the structure (connections and directions) and the parameters(likelihood) using the data. After the structure has been learned the class is only determined by the nodes in the Markov blanket(its parents, its children, and the parents of its children), and all variables given the Markov blanket are discarded.

For the Naive Bayesian Network which is more well-known nowadays, all features are considered as attributes and are independent given the class.

Bayesian networks and naive Bayesian network have their own advantages and disadvantages and we can see the performance comparison(done on 25 data sets mainly from the UCI repository) as depicted below:

We can see that there are some points below the diagonal line representing the Naive Bayes performs better than the Bayesian Network on those datasets and some points above the diagonal line representing the reverse on some other datasets.

Bayesian Network is more complicated than the Naive Bayes but they almost perform equally well, and the reason is that all the datasets on which the Bayesian network performs worse than the Naive Bayes have more than 15 attributes. That's during the structure learning some crucial attributes are discarded.

We can combine the two and add some connections between the features of the Naive Bayes and it becomes the tree augmented Naive Bayes or k-dependence Bayesian classifier.

References:
1. Bayesian Network Classifiers