Change threshold in ReLU in Caffe framework

Question

I'm new to Caffe, and now I need to change the threshold values ​​in the ReLU layers in the convolution neural network. The way I'm currently using to change thresholds is to edit the C ++ source code in caffe/src/caffe/layers/relu_layer.cppand recompile it. However, this will change the threshold value to the specified value each time a ReLU is called. Is there a way to use a different value as a threshold in each ReLU layer on the network? By the way, I am using the pycaffe interface, and I cannot find such a way to do this.

Finally, sorry for my poor English, if there is something obscure, just let me know, I will try to describe it in detail.

Answer 1

If I understand correctly, your "ReLU with threshold" is basically

f(x) = x-threshold if x>threshold, 0 otherwise

You can easily implement it by adding a layer "Bias"that subtracts thresholdfrom the input immediately before the regular layer"ReLU"

Answer 2

Yes, you can. In src/caffe/protoadd the line:

message ReLUParameter {
  ...
  optional float threshold = 3 [default = 0]; #add this line
  ... 
}

and in src/caffe/layers/relu_layer.cpp, make some small modifications like:

template <typename Dtype>
void ReLULayer<Dtype>::Forward_cpu(const vector<Blob<Dtype>*>& bottom,
    const vector<Blob<Dtype>*>& top) {
  ...
  Dtype threshold = this->layer_param_.relu_param().threshold(); //add this line
  for (int i = 0; i < count; ++i) {
    top_data[i] = (bottom_data[i] > threshold) ? (bottom_data[i] - threshold) : 
                  (negative_slope * (bottom_data[i] - threshold));
  }
}

template <typename Dtype>
void ReLULayer<Dtype>::Backward_cpu(const vector<Blob<Dtype>*>& top,
    const vector<bool>& propagate_down,
    const vector<Blob<Dtype>*>& bottom) {
  if (propagate_down[0]) {
    ...
    Dtype threshold = this->layer_param_.relu_param().threshold(); //this line
    for (int i = 0; i < count; ++i) {
      bottom_diff[i] = top_diff[i] * ((bottom_data[i] > threshold)
          + negative_slope * (bottom_data[i] <= threshold));
    }
  }
}

and similarly, the src/caffe/layers/relu_layer.cucode should look like this .

And after compiling yours caffeand pycaffein, net.prototxtyou can write a layer relu, for example:

layer {
  name: "threshold_relu"
  type: "ReLU"
  relu_param: {threshold: 1 #e.g. you want this relu layer to have a threshold 1}
  bottom: "input"
  top: "output"
}