I tried to implement XOR-gate with tensor flow. I managed to implement this, but I do not quite understand why it works. I got help from stackoverflow columns here and here . So both with outputs one hot trueand without one hot true. Here is the network, as I understand it, so that everything is clear.
My Question # 1:
Pay attention to the function RELUand Sigmoid. Why do we need this (in particular, a function RELU)? You can say that to achieve non-linearity. I understand how RELUnon-linearity is achieved. I got the answer here . Now from what I understand, the difference between using RELUand without using RELUis this (see. Figure). [I tested the function tf.nn.relu. The result is:]
Now, if the first function works, why not the second function? From my point of view, it RELUreaches nonlinearity by combining several linear functions. Thus, both linear functions (the top two). If the first reaches nonlinearity, the second should also, right? The question is, without using RELU, why is the network stuck?
XOR
hidden1_neuron = 10
def Network(x, weights, bias):
layer1 = tf.nn.relu(tf.matmul(x, weights['h1']) + bias['h1'])
layer_final = tf.matmul(layer1, weights['out']) + bias['out']
return layer_final
weight = {
'h1' : tf.Variable(tf.random_normal([2, hidden1_neuron])),
'out': tf.Variable(tf.random_normal([hidden1_neuron, 2]))
}
bias = {
'h1' : tf.Variable(tf.random_normal([hidden1_neuron])),
'out': tf.Variable(tf.random_normal([2]))
}
x = tf.placeholder(tf.float32, [None, 2])
y = tf.placeholder(tf.float32, [None, 2])
net = Network(x, weight, bias)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(net, y)
loss = tf.reduce_mean(cross_entropy)
train_op = tf.train.AdamOptimizer(0.2).minimize(loss)
init_op = tf.initialize_all_variables()
xTrain = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
yTrain = np.array([[1, 0], [0, 1], [0, 1], [1, 0]])
with tf.Session() as sess:
sess.run(init_op)
for i in range(5000):
train_data = sess.run(train_op, feed_dict={x: xTrain, y: yTrain})
loss_val = sess.run(loss, feed_dict={x: xTrain, y: yTrain})
if(not(i%500)):
print(loss_val)
result = sess.run(net, feed_dict={x:xTrain})
print(result)
, , XOR . tf.nn.relu, . ?
# 2:
, [ - ]? ( )? , , , - . -. ( , .)
№ 3:
, hidden1_neuron = 10. , 10. 5 . , ?
, , :
2.42076
0.000456363
0.000149548
7.40216e-05
4.34194e-05
2.78939e-05
1.8924e-05
1.33214e-05
9.62602e-06
7.06308e-06
[[ 7.5128479 -7.58900356]
[-5.65254211 5.28509617]
[-6.96340656 6.62380219]
[ 7.26610374 -5.9665451 ]]
, :
1.45679
0.346579
0.346575
0.346575
0.346574
0.346574
0.346574
0.346574
0.346574
0.346574
[[ 15.70696926 -18.21559143]
[ -7.1562047 9.75774956]
[ -0.03214722 -0.03214724]
[ -0.03214722 -0.03214724]]