Working with exception handling and re-queuing in RQ on Heroku

Question

I have a website running on Heroku in Python, and I have a worker as a background process for handling tasks that I don’t want to block the delivery of the web page, and therefore are not suitable for web speakers. To do this, I configured the queue using rq and redid it.

In my process, sometimes user exceptions may occur. For a specific subset of them, instead of letting the job go directly to the crash queue, I want to request it several times. I looked at the exception handlers page on the rq homepage and I don't quite understand some things. In particular, it describes the following method of writing an exception handler:

 def my_handler(job, exc_type, exc_value, traceback): # do custom things here # for example, write the exception info to a DB ... 

Now I plan to do something like:

  from rq import requeue_job def my_handler(job, exc_type, exc_value, traceback): if exec_type == "MyCustomError": job.meta['MyErrorCount'] += 1 job.save() if job.meta['MyErrorCount'] >= 10: return True else: requeue_job(job.id) return False 

Questions:

• What are the objects exc_type , exc_value and traceback ? (for example, is the if exec_type == "MyCustomError" correct?)
• Will my error handler effectively detect if this is a specific error, request these jobs until it works 10 times, and then let it fall to failed ? Will it also allow all other errors to fall to failed ?

Answer 1

Here is my solution

 queues = [] def retry_handler(job, exc_type, exc_value, traceback): # Returning True moves the job to the failed queue (or continue to # the next handler) job.meta.setdefault('failures', 1) job.meta['failures'] += 1 if job.meta['failures'] > 3 or isinstance(exc_type, (LookupError, CorruptImageError)): job.save() return True job.status = Status.QUEUED for queue_ in queues: if queue_.name == job.origin: queue_.enqueue_job(job, timeout=job.timeout) break else: return True # Queue has disappeared, fail job return False # Job is handled. Stop the handler chain. queues.append(Queue(exc_handler=retry_handler)) 

I decided to repeat all the errors three times if some known type of exception was not found. This allows me to respect errors that are understood, for example, if the user was deleted after the task was created, but before the task was completed or if the image was resized, the provided image was no longer found (HTTP 404) or not in a readable format (mainly when I I know that the code will never process the task).

To answer your question: exc_type is a class, exc_value is an exception instance. traceback is useful for logging. If you're interested, check out Sentry. Workers are automatically configured using the Sentry error handler if it uses SENTRY_DSN . Much cleaner than polluting your own db error logs.

Answer 2

• read the sys document for more information.

• False means to stop exception handling, True means to continue and move to the next exception handler on the stack

several exception handlers of the same task, type - type of exception (class), you must fix your code, other errors will return None , interpreted as True , as rq doc says.