What is a good way to format logs?

Question

I am developing an application that includes the need to register all incoming messages that I receive from a Telnet connection. The text is pretty simple, although it can include ANSI tags that provide text color and formatting (16 colors, bold, underline, etc.).

I would like to format my logs to store text with formatting, date / time, and potentially other metadata later. My first thoughts were all XML, but it could affect my ability to quickly write a quick search tool. My current idea is date / time + text in one file with metadata stored in another XML file referenced by line number.

This is a good decision? Also, where and how to store formatting commands? The original ANSI tags will break the normal, but having them in two different files can be inconvenient.

Extras: Thanks to some of the answers so far, although I should mention that in most cases the messages will be person-to-person messages, not system messages. More primitive IRC. Its up to my user to decide later (adding metadata) which messages were important. This is an unprocessed logbook from which logs can be filtered or edited.

Answer 1

My first guess would be to use a logging tool like log4net, which will make formatting much more automatic.

If you are going to go the route of two files (and I agree with Craig that the database is probably the best choice), you can probably save yourself a lot of suffering by having one file that is as sparse as you can make it for faster, faster searches, and one that stores all the information in one place (metadata and data), rather than creating a format for metadata only.

Answer 2

G'day

Definitely register in a flat file and add munge scripts to later enable it in XMl.

The first suggestion is to make sure that all date / time strings are in ISO 8601 format, namely YYYY-MM-DD hh: mm: ss.

Secondly, to make your categories, for example. exception, fatal, error, warning, information, etc. really stand out in your magazines.

Then look at some vim syntax files and create new syntax for your log format so that important log entries really stand out.

It’s not so difficult to take one of the standard syntax files and modify it to process log lines.

NTN.

amuses

Rob

Answer 3

If you submit registration information for future searches and anaylsis, the database might be the best answer.

As for your decision. Flat files do not scale very well where the database scale is much better. I wouldn’t split files either, which just binds the scalability problem. If you need to use a flat file, I would probably try to save the metadata in csv (less overhead) and the data in a series of files indexed by a csv file. Thus, all data does not affect your index file. Just my thoughts.

Answer 4

I'm going to “split the fence” and say that I use a database for all of your analysis / archiving log entries (for example, your Telnet messages). This will give you the benefits of a complete search for text, columns, and easy ways to find data.

Use a flat file (or XML format, since the file should not be too large) for any of your logs such as debugging / critical error logs.

If you have a broken database connection or something went awkward with your table structure, registering in the database will be pointless.

Think about it, if you are looking for a slightly more “lightweight” solution, you can use SQLite to log all your telnet traffic so that you can take advantage of the database structure but also have file accessibility.

With another log4net worship, you can easily accomplish this with the added ADO application.

Answer 5

I'm not sure what you are trying to accomplish. Typically, Telnet is considered a protocol with a character in time, so when you say "incoming messages" do you mean that each character is a message? Or is the entire user session a message?

I will make some assumptions. You have users logging in through telnet, and you want to capture everything that they do during login. Later, you will want to associate the material that they did with this user, and the time and date when they did it. You will need to find it later to find out "who made" rm * "as root?"

I would save each user session as a separate file with a naming convention that includes user registration and timestamp.

eg. 2008_09_08_14_52_07_nidonocu

Inside the file, I would capture every byte received, assuming that they would be mostly text characters.

eg.

ls cd www ls vi index.html /copyright 2007 llllllllllllr8:wq exit 

Write 8-bit ANSI characters to a file. You should be able to use a text editor and grep to perform basic audits and searches. You can use the binary viewer or get more sophisticated information if you really need to read 8-bit data.

Backups, archiving, cleaning, etc. can be performed using common file system tools and scripts.

My apologies if my assumptions are wrong.

- Page Bruce