Neural network for predicting the nth area

Question

I am trying to use a multilayer neural network to predict the nth square.

I have the following training data containing the first 99 squares

1 1 2 4 3 9 4 16 5 25 ... 98 9604 99 9801 

This is the code:

 import numpy as np import neurolab as nl # Load input data text = np.loadtxt('data_sq.txt') # Separate it into datapoints and labels data = text[:, :1] labels = text[:, 1:] # Define a multilayer neural network with 2 hidden layers; # First hidden layer consists of 10 neurons # Second hidden layer consists of 6 neurons # Output layer consists of 1 neuron nn = nl.net.newff([[0, 99]], [10, 6, 1]) # Train the neural network error_progress = nn.train(data, labels, epochs=2000, show=10, goal=0.01) # Run the classifier on test datapoints print('\nTest results:') data_test = [[100], [101]] for item in data_test: print(item, '-->', nn.sim([item])[0]) 

What prints 1 for the 100th and 101st squares:

 Test results: [100] --> [ 1.] [101] --> [ 1.] 

What is the right way to do this?

Answer 1

Following the suggestions and comments of Philip Malchak and Sheani123, I implemented a neural network in a tensor stream to check what happens when we try to teach it to predict (and interpolate) the 2nd square.

Continuous Interval Training

I trained the network on the interval [-7, 7] (taking 300 points inside this interval to make it continuous), and then tested it on the interval [-30, 30]. The activation functions are ReLu, and there are 3 hidden layers on the network, each of which has a size of 50. epochs = 500. The result is shown in the figure below.

Thus, basically, inside (and also close) to the interval [-7, 7], the fit is perfectly perfect, and then it continues more or less linearly outside. It's nice to see that at least at the initial stage, the slope of the network output is trying to "match" the slope x^2 . If we increase the testing interval, the two graphs diverge greatly, as can be seen in the figure below:

Even number training

Finally, if instead I train the network on the set of all even numbers in the interval [-100, 100] and apply it to the set of all integers (even and odd) in this interval, I get:

When training the network to create the image higher, I increased the era to 2500 to get better accuracy. Other parameters remained unchanged. Thus, it seems that interpolation “inside” the training interval works pretty well (perhaps with the exception of the area around 0, where the fit is a little worse).

Here is the code I used for the first drawing:

 import tensorflow as tf import matplotlib.pyplot as plt import numpy as np from tensorflow.python.framework.ops import reset_default_graph #preparing training data train_x=np.linspace(-7,7,300).reshape(-1,1) train_y=train_x**2 #setting network features dimensions=[50,50,50,1] epochs=500 batch_size=5 reset_default_graph() X=tf.placeholder(tf.float32, shape=[None,1]) Y=tf.placeholder(tf.float32, shape=[None,1]) weights=[] biases=[] n_inputs=1 #initializing variables for i,n_outputs in enumerate(dimensions): with tf.variable_scope("layer_{}".format(i)): w=tf.get_variable(name="W",shape=[n_inputs,n_outputs],initializer=tf.random_normal_initializer(mean=0.0,stddev=0.02,seed=42)) b=tf.get_variable(name="b",initializer=tf.zeros_initializer(shape=[n_outputs])) weights.append(w) biases.append(b) n_inputs=n_outputs def forward_pass(X,weights,biases): h=X for i in range(len(weights)): h=tf.add(tf.matmul(h,weights[i]),biases[i]) h=tf.nn.relu(h) return h output_layer=forward_pass(X,weights,biases) cost=tf.reduce_mean(tf.squared_difference(output_layer,Y),1) cost=tf.reduce_sum(cost) optimizer=tf.train.AdamOptimizer(learning_rate=0.01).minimize(cost) with tf.Session() as sess: sess.run(tf.global_variables_initializer()) #train the network for i in range(epochs): idx=np.arange(len(train_x)) np.random.shuffle(idx) for j in range(len(train_x)//batch_size): cur_idx=idx[batch_size*j:batch_size*(j+1)] sess.run(optimizer,feed_dict={X:train_x[cur_idx],Y:train_y[cur_idx]}) #current_cost=sess.run(cost,feed_dict={X:train_x,Y:train_y}) #print(current_cost) #apply the network on the test data test_x=np.linspace(-30,30,300) network_output=sess.run(output_layer,feed_dict={X:test_x.reshape(-1,1)}) plt.plot(test_x,test_x**2,color='r',label='y=x^2') plt.plot(test_x,network_output,color='b',label='network output') plt.legend(loc='center') plt.show() 

Answer 2

Documents checked for neurolab - newff creates NN with sigmoid by default in all neurons. The Sigmoid value is always in the range (-1; 1) , so your output will never leave this range.

The second square (4) is already out of this range, so your code does not match your problem at all.

Try using other functions (I would suggest SoftPlus or ReLU ). They work well with direct transmission networks, allow for reverse engineering (since they are displayed in the whole domain) and have values ​​in the range (0, ∞) , as you need.

Also: the first parameter newff defines the ranges for the input data - you use [0, 99], which matches all the training data, but does not match the values ​​you tried to test (since 100 and 101 are greater than 99). Change this value to something larger, so the values ​​you are testing are not "special" (which means "at the end of the range") - I would suggest something like [-300, 300] .

Also, as Seanny123 stated in a comment, I don't think it will work at all, but with the current setup I can be sure of that. Good luck. Let me know (e.g. in the comments) if you succeed.

Last but not least, what you are trying to do is extrapolation (calculating values ​​from a certain range based on values ​​in that range). NNs are better suited for interpolation (calculating values ​​in a range based on samples from that range) because they should summarize the data used in training. Try to study its squares, for example, every third square (like 1, 16, 49, ...), and then test by asking for the squares of the rest (for example, asking for square 2 or 8).

Answer 3

I am trying to use a multilayer neural network to predict the nth square.

Here is a more general discussion of why NNs are not suitable for this: can neural networks approximate any function given a sufficient number of hidden neurons? As for any extrapolation, as Philippe Malchak noted.