PyBrain: When building a network from scratch, how and where do you create bias?

Question

Following the PyBrain documentation, Creating networks with modules and connections , I create a neural network piecewise (as opposed to using the buildNetwork shortcut). I create a simple three-layer (input, hidden, output) neural network. How to add an offset block?

I assume that I am BiasUnit module, as in:

 b = BiasUnit(name='bias') network.addModule(b) 

Is it correct? Do I need to create a FullConnection object? If so, what should I connect?

Answer 1

The implemented PyBrain is open source, and I have the source code that is in my Python directory. I opened the file C: \ Python27 \ Lib \ site-packages \ pybrain \ tools \ shortcuts.py. Inside this file, I found the buildNetwork function and saw how it adds BiasUnit. The relevant code is here:

 ... n = Network() # linear input layer n.addInputModule(LinearLayer(layers[0], name='in')) # output layer of type 'outclass' n.addOutputModule(opt['outclass'](layers[-1], name='out')) if opt['bias']: # add bias module and connection to out module, if desired n.addModule(BiasUnit(name='bias')) if opt['outputbias']: n.addConnection(FullConnection(n['bias'], n['out'])) # arbitrary number of hidden layers of type 'hiddenclass' for i, num in enumerate(layers[1:-1]): layername = 'hidden%i' % i n.addModule(opt['hiddenclass'](num, name=layername)) if opt['bias']: # also connect all the layers with the bias n.addConnection(FullConnection(n['bias'], n[layername])) # connections between hidden layers ... 

Basically, it seems that he creates a separate BiasUnit and connects it to each hidden layer and, possibly, to the output level.

Answer 2

Here you have a simple example :

 n = RecurrentNetwork() n.addModule(TanhLayer(hsize, name = 'h')) n.addModule(BiasUnit(name = 'bias')) n.addOutputModule(LinearLayer(1, name = 'out')) n.addConnection(FullConnection(n['bias'], n['h'])) n.addConnection(FullConnection(n['h'], n['out'])) n.sortModules() 

Note that BiasUnit connected to TanhLayer , effectively making layer h an offset layer.