What are the advantages of online learning when training neural networks?

Stochastic gradient descent with a batch size of 1 is apparently used to learn from single examples as they arrive, but I don't understand why you would use such a small batch size instead of batching training inputs together.

Batch learning and online learning both have their place. Generally speaking batch learning will train your neural network to a lower residual error level, because the online training can sometimes have one training undo the effect of another. However, online training has a few advantages:

• Online learning schemes learn "faster." In some cases, determining how many samples you need can be difficult, and sample gathering may be costly. Online learning lets you see the progress of your training as the number of samples increases, potentially saving on sample gathering costs once it reaches an acceptable error.

• Online learning schemes can be done during operation, which can be valuable.

• Online learning schemes are more effective for dealing with data sets that are not stationary. It is easier for the batch methods to come up with highly specialized solutions which work well for the test samples, but do not capture a general solution.

StackExchange Computer Science Q#51260, answer score: 8