Limit your Google Cloud Pub / Sub Queue limit

Question

I use the Google Pub / Sub queue to process messages between services. Some of the subscribers connect to the speed limit API.

For example, I click street addresses in a pub / subtopic. I have a Cloud function that subscribes (via push) to this topic and calls an external geocoding service with a speed limit. Ideally, my street addresses can be transferred to the topic without delay, and the topic will save these messages - calling the subscriber with a speed limit.

Is there a way to set such a delay or limit the speed of message distribution? Enlarging the Ack window really does not help: I ​​created this system to prevent long-running functions.

Answer 1

Testing to solve your problem is to use: async.queue

There you have the concurrency attribute, which you can control the speed limit.

 // create a queue object with concurrency 2 var q = async.queue(function(task, callback) { console.log('hello ' + task.name); callback(); }, 2); // assign a callback q.drain = function() { console.log('all items have been processed'); }; // add some items to the queue q.push({name: 'foo'}, function(err) { console.log('finished processing foo'); }); // quoted from async documentation