I have a specific case where the networks are relatively small, and for convergence and generalization issues I have to maintain a small lot size (for example, 256), which leads to hundreds of batches for processing in the era.
Unfortunately, in this case, the calculation of batch loading and loss becomes a bottleneck (as the timeline tool says).
In tensorflow you can write something like this to upload data to gpu:
with tf.device('/gpu:0'): train_data = tf.constant(train_data_numpy)
But if I pass train_data to the Keras Model.predict or Model.fit , I get the following error:
keras/engine/training.pyc in predict(self, x, batch_size, verbose) 1515 f = self.predict_function 1516 return self._predict_loop(f, ins, -> 1517 batch_size=batch_size, verbose=verbose) 1518 1519 def train_on_batch(self, x, y, keras/engine/training.pyc in _predict_loop(self, f, ins, batch_size, verbose) 1129 if verbose == 1: 1130 progbar = Progbar(target=samples) -> 1131 batches = _make_batches(samples, batch_size) 1132 index_array = np.arange(samples) 1133 for batch_index, (batch_start, batch_end) in enumerate(batches): keras/engine/training.pyc in _make_batches(size, batch_size) 368 A list of tuples of array indices. 369 """ --> 370 num_batches = int(np.ceil(size / float(batch_size))) 371 return [(i * batch_size, min(size, (i + 1) * batch_size)) 372 for i in range(0, num_batches)] AttributeError: 'Dimension' object has no attribute 'ceil'
This makes sense, since Keras only expects multi-like arrays and lists of such.
Having said that, I also tried pyCUDA and docked arrays, as they say they are null ... but they cause the following errors:
keras/engine/training.pyc in predict(self, x, batch_size, verbose) 1515 f = self.predict_function 1516 return self._predict_loop(f, ins, -> 1517 batch_size=batch_size, verbose=verbose) 1518 1519 def train_on_batch(self, x, y, keras/engine/training.pyc in _predict_loop(self, f, ins, batch_size, verbose) 1139 ins_batch = _slice_arrays(ins, batch_ids) 1140 -> 1141 batch_outs = f(ins_batch) 1142 if not isinstance(batch_outs, list): 1143 batch_outs = [batch_outs] keras/backend/tensorflow_backend.pyc in __call__(self, inputs) 2266 updated = session.run(self.outputs + [self.updates_op], 2267 feed_dict=feed_dict, -> 2268 **self.session_kwargs) 2269 return updated[:len(self.outputs)] 2270 tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata) 893 try: 894 result = self._run(None, fetches, feed_dict, options_ptr, --> 895 run_metadata_ptr) 896 if run_metadata: 897 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata) 1091 feed_handles[subfeed_t] = subfeed_val 1092 else: -> 1093 np_val = np.asarray(subfeed_val, dtype=subfeed_dtype) 1094 1095 if (not is_tensor_handle_feed and numpy/core/numeric.pyc in asarray(a, dtype, order) 529 530 """ --> 531 return array(a, dtype, copy=False, order=order) 532 533 ValueError: object __array__ method not producing an array
I tried to deal with this problem, but the only reasonable coincidence is the blog post on the Chinese blog, which basically proposes a Keras fix, which is obviously impractical.
I wonder what the correct way to pre-install the entire dataset for gpu for Keras.
Useful information: I am using Keras 2.0.6 with TF 1.3, the upgrade to the 2.0.8 / 1.4 stack is not yet available due to important API changes, but it will definitely be accelerated if it solves this problem.