Why are processes running in the Google App Engine freezing?

Question

We have a cron task that runs every hour on the backend module and creates tasks. The cron task runs queries in the Cloud SQL database, and the tasks make HTTP calls to other servers and also update the database. Usually they work fine, even when thousands of tasks are created, but sometimes it gets stuck and nothing can shed light on the situation in the magazines. For example, yesterday we tracked a cron job when it created several dozen tasks and then stopped, as well as 8 tasks that were also stuck in the queue. When it was obvious that nothing was happening, we started the process a few more times and successfully completed each time.

A day later, the original task was killed with a DeadlineExceededException, and then another 8 other tasks that obviously worked in one instance were killed with the following message: There was a problem with the process that processed this request, causing it to exit. This will likely cause the new process to be used for the next request to your application. If you often see this message, you can throw exceptions during the initialization of your application. (Error code 104)

Until the processes were killed, we did not see any entries in them, and now, when we see them, there are no log entries until DeadlineExceededException, so we have no idea at what point they were stuck. We suspected that there was some lock in the database, but in the following link we see that there was a 10-minute limit for requests, so this would cause the process to crash earlier than one day later: https://cloud.google.com / appengine / docs / java / cloud-sql / # Java_Size_and_access_limits

Our module class and scale configuration:

<instance-class>B4</instance-class>
<basic-scaling>
    <max-instances>11</max-instances>
    <idle-timeout>10m</idle-timeout>
</basic-scaling>

Queue Configuration:

<rate>5/s</rate>
<max-concurrent-requests>100</max-concurrent-requests>
<mode>push</mode>
<retry-parameters>
    <task-retry-limit>5</task-retry-limit>
    <min-backoff-seconds>10</min-backoff-seconds>
    <max-backoff-seconds>200</max-backoff-seconds>
</retry-parameters>

I uploaded several images of trace data for the cron job: http://imgur.com/a/H5wGG . This includes a trace summary and a start / end timeline. No trace data for 8 completed tasks.

?

Answer 1

:

• - cron . , , .
• 2, , -, , , .