When to update neural network model in production?

Question

I have a trained neural network about to go into production, and am wondering about when to update it with new data.

Say, if the "Similar questions" banner when asking a question is recommending other questions with a neural network, trained some time ago, on duplicate questions. Under what circumstances should Stack Exchange update the network?

I tried searching in the literature but did not find anything at a glance.

Are there some best practices/research on when to update the production models?

Answer 1

One of the most significant challenges with neural networks is that they are a black box. There are several techniques that you can utilize to improve them, including providing more training data, but the only way that you can really know how effective your neural net is is to watch it and see how well it performs. Even then, there's no way to know for sure.

The military did a study once on neural networks. The problem was to identify pictures containing tanks. The neural net performed flawlessly on the training data, so they released it to production, where they discovered that it performed no better than statistical chance would predict. So they went back to their training images, examined them again, and realized that all of the pictures containing tanks were taken on cloudy days. The network was identifying clouds, not tanks.

Training neural networks is still more art than science.

Should you update your neural net with more data? That depends. Is it already performing adequately? Do you want it to get smarter as time goes along? Does your neural network adequately generalize, or is it overfitting?

Unfortunately there's no "one size fits all, best practice, this is the way you're grandma used to make cookies" solution when it comes to neural networks. You have to base your training model on factors like testing, simulations, amount of resource available to expend on the problem, and so forth.