A neural network generating incorrect results that are around the average of the results

Question

I am working on a neural network system to perform SED fitting as part of a student project at the University of Western Australia.

I created a set of 20,000 runs through the SED installer known as MAGPHYS . Each run has 42 input values ​​and 32 output values ​​that interest us (the system has more outputs, but we do not need them)

I experimented with the Keras neural network package to create a network to learn this feature.

My current network design uses 4 hidden layers, completely interconnected, with 30 connections between each layer. Each layer uses TanH activation features. I also have a 42-level input layer and a 32-dimensional output layer, both also using TanH activation, for a total of 6 layers.

model = Sequential()
loss = 'mse'
optimiser = SGD(lr=0.01, momentum=0.0, decay=0, nesterov=True)

model.add(Dense(output_dim=30, input_dim=42, init='glorot_uniform', activation='tanh'))

for i in range(0, 4):
    model.add(Dense(output_dim=30, input_dim=30, init='glorot_uniform', activation='tanh'))

model.add(Dense(output_dim=32, input_dim=30, init='glorot_uniform', activation='tanh'))
model.compile(loss=loss, optimizer=optimiser)

I use the minimum / maximum normalization of my input and output data to squash all values ​​between 0 and 1. I use the stochastic gradient descent optimizer and I experimented with various loss functions such as mean squared error, mean absolute error, average absolute percentage error and etc.

, , , , . , , . , , , , , .

( 32 ):

Output   Correct
9.42609868658  =   9.647
9.26345946681  =   9.487
9.43403506231  =   9.522
9.35685760748  =   9.792
9.20564885211  =   9.287
9.39240577382  =   8.002

, 9,2-9,4, .

, , ​​ , , ?

- , ?

Answer 1

, CAFEBABE:

• 42 - . , , , ( ), /. , 20K . , . .

• / , , BatchNormalizing , . , , , , .

• . :

  • ? .
  • , rmsprop adam, .
  • , , , .

• . : ReLU, ELU, PReLU, SReLU. keras.

• , . Dropout, L2/L1

• ( ) , 42 30, , . - , 100 500, 1000.

, , :

# imports 
from sklearn.cross_validation import train_test_split
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.normalization import BatchNormalization
from keras.layers.advanced_activations import ELU     

# data shapes
n_obs, n_feat = 20000, 42
n_hidden = 500 # play with this, bigger tends to give better separability
n_class = 32

# instantiate model
model = Sequential()

# first layer --- input
model.add(Dense(input_dim = n_feat, output_dim = n_hidden))
model.add(BatchNormalization())
model.add(ELU())
model.add(Dropout(p=0.2)) # means that 20% of the nodes are turned off, randomly

# second layer --- hidden
model.add(Dense(input_dim = n_hidden, output_dim = n_hidden))
model.add(BatchNormalization())
model.add(ELU())
model.add(Dropout(p=0.2))

# third layer --- output
model.add(Dense(input_dim = n_hidden, output_dim = n_class))
model.add(BatchNormalization())
model.add(Activation('softmax'))

# configure optimization
model.compile(optimizer = 'rmsprop', loss = 'categorical_crossentropy', metrics = ['accuracy'])

# split your data, so you test it on validation data
X_train, X_test, Y_train, Y_test = train_test_split(data, targets)

# train your model
model.fit(X_train, Y_train, validation_data = (X_test, Y_test))

!