We host our infrastructure on the Google Compute Engine and view Autoscaling for instance groups. We do a lot of batch processing of binary data from the queue. In our case, this means:
- When the worker processes the data, the CPU is always 100%
- When the queue is empty, we want to stop all employees
- Depending on the length of the queue, we want a certain number of employees
However, it’s difficult for me to determine how to automatically scale this in the Google Compute Engine, because they seem to scale based on metrics only for the instance, such as the processor. From the doc:
Not all custom metrics can be used by autoscaler. To select a valid custom metric, the metric must have all of the following Properties:
- The metric must be a metric for each instance.
- The metric must be a valid usage metric, which means that the data from the metric can be used to scale up or down the number of virtual machines.
If I read the documentation correctly, this makes it difficult to use automatic scaling in the global queue.
Backup Solutions