Fine stop work

Question

I have several Gearman workers that work constantly, keeping things like recordings of user page views, etc. Sometimes I update the PHP code used by Gearman workers. To get workers to switch to new code, I kill and restart PHP processes for workers.

Which is better for this? Presumably, I sometimes lose data (although not very important data) when I kill one of these workflows.

Edit: I found an answer that works for me, and posted it below.

Answer 1

Ok, I posted this question, now I think I found a good answer for it.

If you look in the code for Net_Gearman_Worker, you will find that the stopWork function is tracked in the work cycle, and if it returns true, it exits the function.

I have done the following:
Using memcache, I created a cached value, gearman_restarttime, and I use a separate script to set it to the current timestamp whenever I update the site. (I used Memcache, but it could be stored anywhere - a database, a file, or something else).

I extended the Worker class, essentially Net_Gearman_Worker_Foo, and all my workers created this. In the Foo class, I flipped the stopWork function to do the following: first, it checks gearman_restarttime; for the first time, it stores the value in a global variable. From now on, each time comparing the cached value with the global one. If it has changed, stopWork returns true, and the worker exits. Cron checks every minute to make sure all workers are still running, and restarts any remaining workers.

It might be worth setting a timer in stopWork and checking the cache only once every x minutes. In our case, Memcache is fast enough that checking the value every time does not seem to be a problem, but if you use some other system to store the current timestamp, the check will be less common.

Answer 2

Solution 1

---

Typically, I start my workers using the daemon unix utility with the -r flag and let them expire after one job. Your script will end gracefully after each iteration, and the daemon will restart automatically.

Your employees will be obsolete for one job, but it may not be as difficult for you as data loss.

This solution also has the advantage of freeing up memory. You may have memory problems if you are doing large tasks since PHP pre 5.3 has a terrible GC.

Decision 2

---

You can also add a quit function to all of your workers that exits the script. When you want to restart, you simply give the repeater calls to exit with high priority.

Answer 3

function AutoRestart() { static $startTime = time(); if (filemtime(__FILE__) > $startTime) { exit(); } } AutoRestart(); 

Answer 4

Hmm, you could implement code from workers to sometimes check if the source code has been changed, if so, then just kill yourself when they find it appropriate. That is, check while they are in the middle of the assignment, and if the work is very large.

Another way is to implement some kind of interruption, perhaps through the network, to say “stop” whenever you have a chance and restart.

The latter solution helps modify the Gearman source to include this functionality.

Answer 5

http://phpscaling.com/2009/06/23/doing-the-work-elsewhere-sidebar-running-the-worker/

As shown in the previous article, I launched the worker inside the BASH script shell, periodically removing the intervals between tasks for cleaning (or reloading worker-w371)), or if the task is given to it, it can exit with a specific exit code and turn off.

Answer 6

I also looked at this recently (albeit in Perl with Gearman :: XS). My account was the same as yours. Allow long staff to periodically check for a new version and reboot.

My first attempt was that the worker always kept track of how long he checked the latest version of the working version of the script (md5sum will also work). Then, as soon as N seconds have elapsed, between tasks, it will check if a new version is available and reboot (fork () / exec ()). This worked fine, but workers registered for rare jobs could potentially expect hours of work () to return and, therefore, to check the current time.

So, now I set a rather short timeout, waiting for work () to work, so I can check the time more regularly. The PHP interface assumes that you can set this timeout value when registering for a job. I use SIGALRM to start checking for a new version. The perl interface blocks operation (), so the alarm did not start initially. Setting a timeout of 60 seconds ensured the operation of SIGALRM.

Answer 7

If someone was looking for an answer for a worker working perl, this is part of what is for the GearmanX :: Starter library. You can stop workers after completing the current job in two different ways: from the outside by sending the SIGTERM workflow or programmatically by setting a global variable.

Answer 8

Given the fact that the workers are written in PHP, it would be nice to rework them according to a well-known schedule. It can be static time from the moment of start or can be executed after a certain number of tasks.

This essentially kills (not a pun) two birds with one stone. You reduce the potential for memory leakage, and you have a consistent way to determine when your employees will take on potentially new code.

I usually write to employees to report their interval to stdout and / or to the logger, so just check where the worker is.

Answer 9

I ran into this problem and came up with a solution for python 2.7.

I am writing a python script that uses gearman to communicate with other components in the system. The script will have multiple workers, and each worker runs in a separate thread. Workers all receive relay data, they process and store this data in a message queue, and the main thread can pull data from the queue if necessary.

My solution for cleanly closing each employee was to subclass gearman.GearmanWorker and override the work() function:

 from gearman import GearmanWorker POLL_TIMEOUT_IN_SECONDS = 60.0 class StoppableWorker(GearmanWorker): def __init__(self, host_list=None): super(StoppableWorker,self).__init__(host_list=host_list) self._exit_runloop = False # OVERRIDDEN def work(self, poll_timeout=POLL_TIMEOUT_IN_SECONDS): worker_connections = [] continue_working = True def continue_while_connections_alive(any_activity): return self.after_poll(any_activity) while continue_working and not self._exit_runloop: worker_connections = self.establish_worker_connections() continue_working = self.poll_connections_until_stopped( worker_connections, continue_while_connections_alive, timeout=poll_timeout) for current_connection in worker_connections: current_connection.close() self.shutdown() def stopwork(self): self._exit_runloop = True 

Use it in the same way as GearmanWorker. When it exits the script, call the stopwork() function. It will not stop right away - it may take up to poll_timeout seconds before it leaves the run loop.

There can be several smart ways to call the stopwork() function. In my case, I create a temporary relay client in the main thread. For the worker I am trying to disable, I send a special STOP command through the relay server. When a worker receives this message, he knows that it is closed.

Hope this helps!

Answer 10

It fits perfectly into your continuous integration system. I hope you have it or you should get it soon :-)

When checking new code, it is automatically created and deployed to the server. As part of the build script, you kill all the workers and launch new ones.

Answer 11

I use the following code that supports both Ctrl-C and kill -TERM . By default, supervisor sends a TERM signal if it has not changed the signal= setting. In PHP 5.3+, declare(ticks = 1) deprecated, use pcntl_signal_dispatch() instead.

 $terminate = false; pcntl_signal(SIGINT, function() use (&$terminate) { $terminate = true; }); pcntl_signal(SIGTERM, function() use (&$terminate) { $terminate = true; }); $worker = new GearmanWorker(); $worker->addOptions(GEARMAN_WORKER_NON_BLOCKING); $worker->setTimeout(1000); $worker->addServer('127.0.0.1', 4730); $worker->addFunction('reverse', function(GearmanJob $job) { return strrev($job->workload()); }); $count = 500 + rand(0, 100); // rand to prevent multple workers restart at same time for($i = 0; $i < $count; $i++) { if ( $terminate ) { break; } else { pcntl_signal_dispatch(); } $worker->work(); if ( $terminate ) { break; } else { pcntl_signal_dispatch(); } if ( GEARMAN_SUCCESS == $worker->returnCode() ) { continue; } if ( GEARMAN_IO_WAIT != $worker->returnCode() && GEARMAN_NO_JOBS != $worker->returnCode() ) { $e = new ErrorException($worker->error(), $worker->returnCode()); // log exception break; } $worker->wait(); } $worker->unregisterAll(); 

Answer 12

I use gearmadmin to check if there are any jobs. I used the admin API to create the user interface. When jobs sit idle, there is no harm in killing them.