What is the best compression library for very small amounts of data (3-4 kilobytes?)

Question

I am working on a game engine that freely departs from Quake 2, adding some things, such as script effects (allowing the server to specify special effects for the client in detail, instead of having only a limited number of hard-coded effects that the client is capable of.) This is a compromise of efficiency Networks for flexibility.

I hit an interesting barrier. See, The maximum packet size is 2800 bytes, and only one can go to each client per frame.

Here is a script to make the effect of "sparks" (may be good for bullet sparks, electric shocks, etc.). http://pastebin.com/m7acdf519 (If you don’t understand this, don’t sweat, this is the special syntax I made and not related to the question I ask.)

I did my best to reduce the size of this script. I even reduced the variable names to separate letters. But the result is exactly 405 bytes. This means that you can place no more than 6 of them per frame. I also mean a few server-side changes that could shave it another 12 and change the protocol, which could save another 6. Although the savings will vary depending on which script you are working with.

However, out of these 387 bytes, I believe that only 41 will be unique between multiple applications of the effect. In other words, this is the main candidate for compression.

It so happened that R1Q2 (Quake 2 backward compatible engine with advanced network protocol) has a Zlib compression code. I could pick up this code, or at least keep a close eye on it.

But is Zlib necessarily the best choice here? I can imagine at least one alternative to LZMA, and it would be easier.

Requirements:

• It should be very fast (it should have very low performance when starting more than 100 times per second.)
• You need to copy as much data as possible in 2800 bytes.
• Small metadata size
• GPL compatible

Zlib looks good, but is there anything better? Keep in mind that none of these codes have yet been merged, so there are many opportunities for experimentation.

Thanks, -Max

EDIT: Thanks to those who suggested compiling the scripts into bytecode. I had to clearly understand this - yes, I do it. If you like, you can view the corresponding source code on my website, although it is still not a “lure”.
This is the server side code:
Lua: http://meliaserlow.dyndns.tv:8000/alienarena/lua_source/lua/scriptedfx.lua
C: http://meliaserlow.dyndns.tv:8000/alienarena/lua_source/game/g_scriptedfx.c
For a specific example of the script that I posted, it gets a source of 1172 bytes to 405 bytes - still not small enough. (Keep in mind that I want as many as possible in 2800 bytes!)

EDIT2: There is no guarantee that any given package will arrive. Each package should contain a "state of the world", without relying on the information transmitted in previous packages. Typically, these scripts will be used to transmit "eye candy." If there is no room for one, it drops out of the package and it does not really matter. But if too much falls out, things begin to look strange visually, and this is undesirable.

Answer 1

FINAL UPDATE: The two libraries look roughly the same. Zlib provides about 20% better compression, while the LZO decoding speed is about twice as fast, but the performance is too low, almost negligible. This is my last answer. Thanks for all the other answers and comments!

UPDATE:. After introducing LZO compression and viewing only more noticeable performance, it is clear that my own code is to blame for success (the number of effect scenarios for each package has increased significantly, the translator effect gets much more.) I would like to humbly apologize for scrambling to shift the blame and I hope that there are no difficulties. I will do some profiling, and then maybe I can get some numbers that will be more useful to someone else.

ORIGINAL MAIL:

OK, I finally got to writing code for this. I started with Zlib, here are my first conclusions.

Zlib compression is insanely great. It reliably reduces packets, say, 8.5 kib to, say, 750 bytes or less, even when compressed with Z_BEST_SPEED (instead of Z_DEFAULT_COMPRESSION.) The compression time is also pretty good.

However, I had no idea that the speed of decompression could not even be so bad. I have no actual numbers, but at least it should take 1/8 of a second per packet! (Core2Duo T550 @ 1.83 GHz.) Absolutely unacceptable.

From what I heard, LZMA is a compromise between worse performance and better compression compared to Zlib. Since the Zlib compression is already full and its performance is already incredibly poor, LZMA is now not visible on the table.

If the LZO decompression time is as good as claimed, then this is what I will use. I think that in the end, the server will still be able to send Zlib packets in extreme cases, but clients can be configured to ignore them, and this will be the default.

Answer 2

LZO can be a good candidate for this.

Answer 3

zlib can be a good candidate - the license is very good, it works quickly, and its authors say that it has very little overhead and overhead is what is used for small amounts of data.

Answer 4

you should look at OpenTNL and adapt some of the methods that they use there, for example, the concept of network strings

Answer 5

I would like to use the most significant bit of each character, which currently disappears in vain, shifting groups of 9 bytes to the left, you fit in 8 bytes.

You can go further and map the characters to a small space - can you bring them up to 6 bits (i.e. having only 64 real characters), for example, not allowing uppercase letters and subtracting 0x20 from each character (so that the space becomes the value 0 )

You can go further by matching the frequency of each character and doing Huffman type compression to reduce the number of bits of the number of each character.

I suspect that there are no algorithms that will save data better, which is generally the case, as a result of the message you made alrady, there is essentially no redundancy in the message.

Answer 6

How about sending a binary representation of your script?

So, I think of the lines of the abstract syntax tree with each procedure that has an identifier.

This means that the gain in formatting on clients arises from a one-time parsing and a decrease in size due to the removal of method names.