A CNN in Python WITHOUT frameworks

Here's some code that I've written for implementing a Convolutional Neural Network for recognising handwritten digits from the MNIST dataset over the last two days (after a lot of research into figuring out how to convert mathematical equations into code).

```
""" Convolutional Neural Network """

import numpy as np
import sklearn.datasets
import random
import math
from skimage.measure import block_reduce
from scipy.signal import convolve
import time

def reLU(z): # activation function
return z * (z > 0)

""" ------------------------------------------------------------------------------- """

class ConvPoolLayer:
def __init__(self, in_dim, filter_dim, pool_dim=None, conv_stride=1):
self.in_dim = in_dim
self.out_dim = (filter_dim[0], int(round(((0.0 + in_dim[-2] - filter_dim[-2]) / conv_stride + 1) / pool_dim[-2])), \
int(round(((0.0 + in_dim[-1] - filter_dim[-1]) / conv_stride + 1) / pool_dim[-1]))) \
if pool_dim \
else \
(num_filters, ((in_dim[-2] - filter_dim[-2]) / conv_stride + 1), \
((in_dim[-1] - filter_dim[-1]) / conv_stride + 1) )

self.filter_dim = filter_dim
self.pool_dim = pool_dim

self.W = np.random.randn(filter_dim) np.sqrt(2.0 / (sum(filter_dim))).astype(np.float32)
self.B = np.zeros(((in_dim[-1] - filter_dim[-1]) / conv_stride + 1, 1)).astype(np.float32)

def feedforward(self, x, W, b, step):
self.x = x.reshape(step, self.in_dim[-2], self.in_dim[-1])
activation = reLU(np.array([convolve(self.x, w, mode='valid') for w in W]) + b.reshape(1, -1, 1))

if self.pool_dim:
return block_reduce(activation, block_size=tuple([1] + list(self.pool_dim)), func=np.max)
else:
return activation

def backpropagate(self, delta, W, index):
delta = delta.reshape(len(W), 1, int((np.prod(delta.shape) // len(W)) ** 0.5), -1)

Disclaimer: I know practically nothing about ML or neural networks.

The big problem with this program is readability. There are no docstrings or comments, so even somebody who knows about ML would have difficulty using this. for example, what arguments should I pass to the ConvPoolLayer constructor? What does reLU(z) represent? And so on.

Particularly when the code is in a specialist area, it’s essential to have comments explaining why the code was written this way – what’s it trying to do, what concepts does the code map to.
This will make it much easier for other people to follow – including you, in six months time!

And some more specific observations:

-
Run a PEP 8 linter. There’s a bunch of little PEP 8 violations – line length, whitespace, and so on – a linting tool like flake8 can help you spot those. Makes your code look more like other Python, and so easier for others to read.

-
Use new-style classes. If you’re using Python 2, your classes should all subclass from object. This comes with a bunch of minor benefits and is generally good practice. See the Python Wiki for more background.

-
Don’t skimp on variable names. Lots of your code uses one or two-letter variable names. That hurts readability, and can make it harder to search for a variable’s use in code. Longer, more expressive names are almost always better – use them!

-
Use collections.namedtuple for multi-part arguments. I’m guessing based on this ML tutorial that the arguments for ConvPoolLayer are multi-part. For example, the filter_dim argument should have four components:

• number of filters

• number of input feature maps

• filter height

• filter width

Right now, you’re accessing those components by numerical index, which isn’t great for readability. If you created a namedtuple to represent a filter shape, you’d be able to look up those properties by name. For example:

from collections import namedtuple

FilterDim = namedtuple('filterdim',
                       ['num_filters', 'num_maps', 'height', 'width'])

foo = FilterDim(5, 5, 10, 3)
foo.num_maps  # 5

This would significantly improve the readability of your code.

-
The __init__ method of ConvPoolLayer.

• It uses a num_filters variable that doesn't seem to be defined anywhere.

-
I’m not a big fan of the foo = bar if condition else baz ternary operator in Python, and in this case it should definitely be split over multiple lines. I’d also suggest breaking the components of the tuple over multiple lines, to make it easier to see where one ends and the next begins. For example:

if pool_dim:
    self.out_dim = (
        filter_dim[0],
        int(round(((0.0 + in_dim[-2] - filter_dim[-2]) / conv_stride + 1) / pool_dim[-2])),
        int(round(((0.0 + in_dim[-1] - filter_dim[-1]) / conv_stride + 1) / pool_dim[-1]))
    )
else:
    self.out_dim = (
        num_filters,
        (in_dim[-2] - filter_dim[-2]) / conv_stride + 1,
        (in_dim[-1] - filter_dim[-1]) / conv_stride + 1
    )

This makes it easier to read, and easier to see the similarities between different arguments. And as above, you should probably consider defining a named tuple for this stuff.

-
Make use of enumerate. When you need to loop over both the index and element of a list, using enumerate() is cleaner than doing one or the other. For example, this snippet:

for i in range(len(targets)):
    delta, dw, db = self.layers[-1].backpropagate(
        inputs[i].reshape(-1, 1),
        targets[i],
        self.W[-1],
        index=i)

can become slightly neater:

for idx, target in enumerate(targets):
    delta, dw, db = self.layers[-1].backpropagate(
        inputs[idx].reshape(-1, 1),
        target,
        self.W[-1],
        index=idx)

-
As well as +=, you have -=. Replace:

self.W += -self.learning_rate * weight_gradients

with

self.W -= self.learning_rate * weight_gradients

StackExchange Code Review Q#133251, answer score: 6