Python, twistedmatrix ... socket registration

Question

I need help to evaluate what the correct twisted-matrix application design will be. Or any url to help do this.

• background: I now use the logging capabilities included in twistedmatrix with FileLogObserver and a custom DailyLogFile to move and save data to the file system and for further analysis.

Now I am going to use many different applications that will perform their tasks and send log messages in the form of time events in many files, twisted logs (files with different files). So my events are composed of (timestamp, data)

I need to somehow read each of these registered messages (timestamp, data) in real time and centralize the grouping of events by time, that is, have some calculations and group by all events that have exactly the same time stamp.

Am I entitled to assume that I can do this using twisted log functions?

how would you lay it in a simple way?

my current thought was to create a twisted socket logger and duplicate the registration process, so that each event is sent to the file logger (I need a separate history), but also through the socket for the recipient (now I also need in real time), if I am correct digging this way, does anyone have a skeleton for such a twisted socket logger? How can I knit two registrars from twisted ones?

for the other part, what should be the correct registration receiver?

since I will also distribute calculated / aggregated data through the lighstreamer server using the diagram below, there may be some difficulties that I might not have seen, which would require some streaming mecanism or other mecanism to avoid a blocking network call?

 feed = socket.socket(socket.AF_INET, socket.SOCK_STREAM) feed.connect(("localhost",MYPORT)) feed.send(mytimestamp, myeventdata) 

as a figure, it can be schematized as: s (almost all in real time)

 producerTimedEventLog1 --->| producerTimedEventLog2 --->| ... |---> loggerReceiverComputingData ---> lighstreamer process ---> mozilla or whatever webclient ... | producerTimedEventLognN--->| 

each producerTimedEventLognN is also written to the file.

I'm interested in all the useful ideas :)

Best wishes

Answer 1

I realized that several computers (independent computers) are involved in your system, right? Then, first of all, you need to make sure that the clocks of the various machines that send events are synchronized to a certain extent. However, since there will always be a time synchronization error, you must decide what you mean by "all events that have exactly the same timestamp." From my point of view, you need to define an arbitrary temporary delta value. Then you simply identify all events with timestamps during that time to happen "at the same time."

Secondly, if I understood correctly, you want the "producers" to send their events to the "collector" via TCP. This raises the question of whether you already have a specific communication protocol, on top of which you can simply add a message about events with a time stamp, or if you need to create an independent communication level (a separate TCP socket in the collector).

In any case, you may need to create a simple event class that will store the event message (log message and any required data), as well as the timestamp of the creation. Manufacturers create instances of this class. You can then use pickle to send a specific event through a byte stream (TCP connection) to the collector. The collector can decompose and process it.

Remember that the resolution of time.time() on Linux is higher than on Windows.

Answer 2

You can add PythonLoggingObserver besides your FileLogObserver . Then configure python logger to send messages to syslog. Syslog can be configured to send messages to a central server. On a central server, you can save messages to a log file and have a different application reading log file with LogReader .

You will also need NTP for clock synchronization.