What is the preferred way to write my linux daemons?

Question

Hello to all

I have a PHP site that should use some cached data (e.g. stored in Memcache). The data must be stored in the cache by daemons that receive it from web services, and some of it must be stored on the MySQL server.

Demons should do the following:

• Get exchange rates, analyze them and save them in a database, as well as in two separate memcache on separate machines.
• Get financial performance and save them in separated memcache.
• Get large XML data and save it in two separate memcache.

I can write these daemons in C / C ++ / Perl / PHP / Python.

I have to decide which language / script I should choose to implement these daemons. The advantage of using PHP for this is that I can use the API used by the website application itself. Another advantage is that PHP is easy, and everyone knows this, so I won’t be attached to supporting these daemons, but on the other hand, PHP is slower and consumes a lot more resources.

The main disadvantage of using a language other than PHP is that it is more difficult to maintain code written in C / C ++ / Perl. Currently, I believe that this is not often done with C / C ++ / Perl. Am I wrong in saying this?

What would you recommend me to do in this case?

Answer 1

Perl and Python are the default answers for writing such scripts. But it doesn’t matter (a lot) which language you use if you write good code. The more important is how you handle your script when it crashes.

Ultimately, you can see that your scripts rarely appear for arbitrary reasons, and you might not need to debug a script because it usually does a fair job and it would be hard to find where it went wrong.

I have few perl scripts that do the same thing you do. it was a difficult task for me to make sure that my scripts did not fail for a long time because I did not want to miss a bunch of live streaming data.

And for this I used monit . Great tool.

Answer 2

PHP is probably the best choice for simplicity / code reuse.

PEAR daemon
Creating daemons in php

EDIT
From what I can say by simply passing data, this is not a problem to worry about. And about the use of resources, just make sure that max_memory is not exhausted (using streaming, perhaps, or configure a lot). Interruption and log operations that take too much time. Connect to the database in a loop when the SQL operation fails, etc.

NOTE CAUTION
Programming a daemon is complicated, and a lot can go wrong. Consider all points of failure.

Also note that Perl is much more versed in daemons than PHP. I left c / C ++ since performance (the bandwidth around) is not a problem, and programming the daemon is quite complicated, like this, why add worries to memory leaks, segfaults, etc.?

Answer 3

The best practice is to use any technology that you know best. You will:

• implement solution faster
• better debug the problems you face
• it’s easier to evaluate libs (or even find out about them) that can offload some of the work for you.
• have an easier time to maintain and extend the code.

Actually, the speed and use of resources will be relatively unimportant if you really do not have real performance requirements.

Answer 4

In short: I would use Python.

more: I tried PHP in cli mode, I experienced a lot of memory leaks , of course, because of bad PHP libraries or PHP libraries that never were, although for another thing, than they die quickly in web request mode (for example, I'm suspicious of PDO).

In the python world, I recently saw a piece of code from shinken , this is a good nagios, rewriting like python daemons, very smart. See http://www.shinken-monitoring.org/the-global-architecture/ and http://www.shinken-monitoring.org/wiki/official/development-hackingcode . As a monitoring tool, you will surely find there very good ideas for some demon reflecting tasks.

Now, can I make an offer? Why not use Shinken or Centreon as a scheduler for data collection tasks? (And maybe soon that I hope Centeron is currently running with a shinken engine instead of a nagios engine)? This can be useful for detecting changes in external data, outputting in samples, etc.

Then for the tasks that need to be performed (data sampling, data conversion, data storage, etc.), this is an ETL job. One of the good open source tools is Talend ETL ( Java ). There are some planning and monitoring tools for Talend , but not open source (sort-of-open-source-where-you-must-pay-a-license). But adding an external scheduler such as Nagios for tasks should be easy (I hope). You will need to verify that memcached is available as a storage engine for Talend ETL or your plugin code.

So this means that instead of language you can think of tools . Or not, depending on the complexity you can assume, each tool adds its own complexity. However, if you want to rebuild everything from scratch, python is fast efficient.

Answer 5

You must use the same language as the rest of your application. This way you can more easily use code and developer skills.

However, as others have noted, PHP is bad for long-running daemons because it handles memory in a way that can leak.

So, I ran these tasks in the "cron" task, which was periodically (reloaded), but make sure that you do not run more copies of the tasks than you plan.

Cron tasks are more reliable than daemons.

• The cron task, which is not running and completing, will start again the next time it is scheduled.
• The cron task, which contains memory leaks, will free memory when it finishes running.
• The cron task, which has its own software (libraries, etc.), automatically captures new versions on subsequent launch without much effort.
• cron already provides startup / shutdown scripts that your Ops team can use to manage it, so you don’t need to rewrite them. Your Ops team already knows how to manage cron and can comment on crontab entries if they want to temporarily disable it.