How to use Keras TensorBoard callback to find a grid

Question

I am using Keras TensorBoard callback. I would like to start a grid search and visualize the results of each individual model in a tensor board. The problem is that all the results of different runs are combined together, and the loss schedule is a mess:

How can I rename each run to have something similar to this:

Here's the grid search code:

df = pd.read_csv('data/prepared_example.csv') df = time_series.create_index(df, datetime_index='DATE', other_index_list=['ITEM', 'AREA']) target = ['D'] attributes = ['S', 'C', 'D-10','D-9', 'D-8', 'D-7', 'D-6', 'D-5', 'D-4', 'D-3', 'D-2', 'D-1'] input_dim = len(attributes) output_dim = len(target) x = df[attributes] y = df[target] param_grid = {'epochs': [10, 20, 50], 'batch_size': [10], 'neurons': [[10, 10, 10]], 'dropout': [[0.0, 0.0], [0.2, 0.2]], 'lr': [0.1]} estimator = KerasRegressor(build_fn=create_3_layers_model, input_dim=input_dim, output_dim=output_dim) tbCallBack = TensorBoard(log_dir='./Graph', histogram_freq=0, write_graph=True, write_images=False) grid = GridSearchCV(estimator=estimator, param_grid=param_grid, n_jobs=-1, scoring=bug_fix_score, cv=3, verbose=0, fit_params={'callbacks': [tbCallBack]}) grid_result = grid.fit(x.as_matrix(), y.as_matrix()) 

Answer 1

I don’t think there is a way to pass the parameter "per turn" to GridSearchCV . Perhaps the easiest approach would be to subclass KerasRegressor to do what you want.

 class KerasRegressorTB(KerasRegressor): def __init__(self, *args, **kwargs): super(KerasRegressorTB, self).__init__(*args, **kwargs) def fit(self, x, y, log_dir=None, **kwargs): cbs = None if log_dir is not None: params = self.get_params() conf = ",".join("{}={}".format(k, params[k]) for k in sorted(params)) conf_dir = os.path.join(log_dir, conf) cbs = [TensorBoard(log_dir=conf_dir, histogram_freq=0, write_graph=True, write_images=False)] super(KerasRegressorTB, self).fit(x, y, callbacks=cbs, **kwargs) 

You would use it like:

 # ... estimator = KerasRegressorTB(build_fn=create_3_layers_model, input_dim=input_dim, output_dim=output_dim) #... grid = GridSearchCV(estimator=estimator, param_grid=param_grid, n_jobs=1, scoring=bug_fix_score, cv=2, verbose=0, fit_params={'log_dir': './Graph'}) grid_result = grid.fit(x.as_matrix(), y.as_matrix()) 

Update:

Since GridSearchCV runs the same model (i.e., the same parameter configuration) more than once due to cross-validation, the previous code ultimately puts several traces in each run. Looking at the source ( here and here ), there seems to be no way to get the “current split identifier”. At the same time, you should not just check for the existence of existing folders and add hooks as needed, because the tasks are being performed (perhaps at least, although I'm not sure if this is the case with Keras / TF) in parallel. You can try something like this:

 import itertools import os class KerasRegressorTB(KerasRegressor): def __init__(self, *args, **kwargs): super(KerasRegressorTB, self).__init__(*args, **kwargs) def fit(self, x, y, log_dir=None, **kwargs): cbs = None if log_dir is not None: # Make sure the base log directory exists try: os.makedirs(log_dir) except OSError: pass params = self.get_params() conf = ",".join("{}={}".format(k, params[k]) for k in sorted(params)) conf_dir_base = os.path.join(log_dir, conf) # Find a new directory to place the logs for i in itertools.count(): try: conf_dir = "{}_split-{}".format(conf_dir_base, i) os.makedirs(conf_dir) break except OSError: pass cbs = [TensorBoard(log_dir=conf_dir, histogram_freq=0, write_graph=True, write_images=False)] super(KerasRegressorTB, self).fit(x, y, callbacks=cbs, **kwargs) 

I use os calls for compatibility with Python 2, but if you use Python 3, you can consider the more convenient pathlib module to handle the path and directory.

Note. I forgot to mention this before, but just in case, note that passing write_graph=True will register a graph for a run, which, depending on your model, can mean a lot (relatively speaking) of this space. The same goes for write_images , although I do not know how much space is required for this function.

Answer 2

It's simple, just save the logs to separate directories with an integrated parameter string as the directory name:

Here is an example of using a date as the name of a run:

 from datetime import datetime datetime_str = ('{date:%Y-%m-%d-%H:%M:%S}'.format(date=datetime.now())) callbacks = [ ModelCheckpoint(model_filepath, monitor='val_loss', save_best_only=True, verbose=0), TensorBoard(log_dir='./logs/'+datetime_str, histogram_freq=0, write_graph=True, write_images=True), ] history = model.fit_generator( generator=generator.batch_generator(is_train=True), epochs=config.N_EPOCHS, steps_per_epoch=100, validation_data=generator.batch_generator(is_train=False), validation_steps=10, verbose=1, shuffle=False, callbacks=callbacks)