Spray containers stop working after 30 minutes

I have a problem with Aquarium and splash. They stop working 30 minutes after the start. The number of pages to download is 50K-80K. I did a cron job to automatically reload every 10 minutes, every Splash container, but it didn’t work. How can I fix this?

Here is a screenshot enter image description here and statistics from HAProxy enter image description here Here is the Splash configuration

splash0: image: scrapinghub/splash:3.0 command: --max-timeout 3600 --slots 150 --maxrss 1000 --verbosity 5 logging: driver: "none" expose: - 8050 mem_limit: 1000m memswap_limit: 1000m restart: always 

And HAProxy

  backend splash-cluster option httpchk GET / balance leastconn # try another instance when connection is dropped retries 2 option redispatch server splash-0 splash0:8050 check maxconn 150 inter 2s fall 10 observe layer4 backend splash-misc balance roundrobin server splash-0 splash0:8050 check fall 15 

UPDATE 1 Here's a script to reload

  #!/bin/sh echo "BEGIN" >> restart.log for index in 0 1 2 3 4 5 6 7 do docker-compose restart splash$index echo "Restarted: Splash$index. Time: $(date '+%A %W %Y %X')" >> restart.log sleep 1 done echo "END" >> restart.log 

And it should work because I have the logs in the Cronjob file:

 cd /var/www/project/aquarium && sh splash_restart.sh 
+5
source share

Source: https://habr.com/ru/post/1275637/


All Articles