Should activation function be monotonic in neural networks?

A lot of activation functions in neural networks (sigmoid, tanh, softmax) are monotonic, continuous and differentiable (except of may be a couple of points, where derivative does not exist).

I understand the reason for continuity and differentiability, but can not really understand a reason for monotonisity.

During the training phase, backpropagation informs each neuron how much it should influence each neuron in the next layer. If the activation function isn't monotonic then increasing the neuron's weight might cause it to have less influence, the opposite of what was intended. The result would be choatic behavior during training, with the network unlikely to converge to a state that yields an accurate classifier.

StackExchange Computer Science Q#45281, answer score: 10