Python neural network: arbitrary number of hidden nodes

I'm trying to write a neural network that only requires the user to specify the dimensionality of the network. Concretely, the user might define a network like this:

nn = NN([2, 10, 20, 15, 2]) # 2 input, 2 output, 10 in hidden 1, 20 in hidden 2...

To do this, I'm trying to adapt some basic code. Please let me know what improvements can be made to improve readability (for example, I've considered cleaning up the distinction between wDims (weight dimensions) and layer dims because these variables seem redundant).

I would also appreciate any tips on how to implement a neural network as a graph. I've tried this but can't agree on what classes need to be defined or even how the graph would be stored (as a python dictionary?). Basically, some suggestions relating to good representation would be much appreciated.

First, auxiliary methods the network uses:

import random, math

random.seed(0)

def r_matrix(m, n, a = -0.5, b = 0.5):
    return [[random.uniform(a,b) for j in range(n)] for i in range(m)]

def sigmoid(x):
    return 1.0/ (1.0 + math.exp(-x))

def d_sigmoid(y):
    return y * (1.0 - y)

Definition and construction of the network:

class NN:

    def __init__(self, dims):

        self.dims    = dims
        self.nO      = self.dims[-1]
        self.nI      = self.dims[0]
        self.nLayers = len(self.dims)
        self.wDims   = [ (self.dims[i-1], self.dims[i])\
                         for i in range(1, self.nLayers) ]

        self.nWeights = len(self.wDims)

        self.__initNeurons()
        self.__initWeights()

    def __initWeights(self):

        self.weights = [0.0] * self.nWeights

        for i in range(self.nWeights):
            n_in, n_out = self.wDims[i]
            self.weights[i] = r_matrix(n_in, n_out)

    def __initNeurons(self):

        self.layers = [0.0] * self.nLayers

        for i in range(self.nLayers):
            self.layers[i] = [0.0] * self.dims[i]

Implementation of back propagation and forwar

1. Comments on your code

-
There's no documentation! What do these functions do? How do I call them? If someone has to maintain this code in a couple of years' time, how will they know what to do?

-
alpha should be a property of the class (and thus an optional argument to the constructor), not an optional argument to the __updateWeights method (where the user has no way to adjust it).

-
You don't need to write \ at the end of a line if you're in the middle of a parenthesis (because the statement can't end until the parenthesis is closed). So there's no need for the \ here:

self.wDims   = [ (self.dims[i-1], self.dims[i])\
                 for i in range(1, self.nLayers) ]

nor in the definitions of t0 and so on.

-
The names could use a lot of work. NN should be NeuralNetwork. MAX should be something like rounds. dims should be something like layer_sizes.

-
You use lots of private method names (starting with __). Why do you do that? The purpose of private names in Python is to "avoid name clashes of names with names defined by subclasses" but that's obviously not necessary here. All that the private names achieve here is to make your code a bit harder to debug:

>>> network = NN((1,1))
>>> network.__backProp(0, [0.5])
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'NN' object has no attribute '__backProp'
>>> network._NN__backProp(0, [0.5])
[0.0]

If you want to indicate that a method is for internal use by a class, then the convention is to prefix it with a single underscore.

-
Generally in Python you should prefer iterating over sequences rather than over indexes. So instead of:

self.wDims   = [ (self.dims[i-1], self.dims[i])\
                 for i in range(1, self.nLayers) ]

you should write something like:

self.wDims = list(zip(self.dims[:-1], self.dims[1:]))

But in practice you'd do better just to drop wDims array altogether, since it just contains the same information as the dims aray. Whenever you write:

n_in, n_out = self.wDims[i]

you could write instead:

n_in, n_out = self.dims[i:i+2]

which seems clearer to me.

-
In the train method, surely you should pass in MAX as a parameter?

-
Instead of looping using while:

i = 0
while i < MAX:
    # ...
    i += 1

prefer looping using for:

for i in range(MAX):
    # ...

(But actually here you don't use i in the loop body, so it would be conventional to write _ instead.)

-
Instead of:

for t in T:
    x, y = t
    # ...

write:

for x, y in T:
    # ...

-
The array layers is not actually a permanent property of the neural networks. It's only used temporarily in feedForward and backPropLearn. And in fact only one layer is really used at a time. It would be better for layer to be a local variable in these methods.

-
The random seed used to initialize the weights should surely be something that you choose each time you create an instance, not just once.

1. Rewriting using NumPy

This code would benefit enormously from using NumPy, a library for fast numerical array operations.

-
Your function r_matrix could be replaced with numpy.random.uniform. Here I create a 4×4 array of random numbers chosen uniformly in the half-open range [2, 3):

>>> numpy.random.uniform(2, 3, (4, 4))
array([[ 2.95552338,  2.75158213,  2.22088904,  2.95417241],
       [ 2.59129035,  2.29089095,  2.16007186,  2.64646486],
       [ 2.39729966,  2.96208642,  2.12305994,  2.68911969],
       [ 2.64394815,  2.21609217,  2.69556204,  2.35376118]])

so the whole of your __initWeights function could become:

self.weights = [numpy.random.uniform(-0.5, 0.5, size)
                for size in zip(self.dims[:-1], self.dims[1:])]

-
Similarly, the whole of your __initNeurons function could become:

self.layers = [numpy.zeros((size,)) for size in self.dims]

But as explained in §1.9 above, we don't actually want self.layers, so __initNeurons can simply be omitted.

-
The _activateLayer method multiplies the vector in self.layers[i-1] by the matrix self.weights[i-1] and then applies the sigmoid function to each element in the result. So in NumPy, _activateLayer becomes:

def _activateLayer(self, i):
    self.layers[i] = sigmoid(numpy.dot(self.layers[i-1], self.weights[i-1]))

We don't even need the check on the length of the input any more, because if we pass in an input array of the wrong length, we'll get:

ValueError: matrices are not aligned

-
Similarly, the _backProp method multiplies the vector delta by the transpose of the matrix self.weights[i], and then applies the d_sigmoid function to each element. So in Numpy this becomes:

def _backprop2(self, i, delta):
    return d_sigmoid(self.layers[i]) * numpy.dot(delta, self.weights[i].T)

Similarly, the _updateWeights method becomes:

```
def _updateWeights(self, i, delta, alpha =

StackExchange Code Review Q#37864, answer score: 6