Optimal loading of data file format on the game console

Question

I need to load large models and other structured binary data onto an older game console on a CD as efficiently as possible. What is the best way to do this? Data will be exported from a Python application. This is a rather complicated hobby project.

Requierements:

• Do not rely on a fully standard STL - I could use uSTL.
• as little as possible overhead. Strive for a solution so well. that it can be used on the original Playstation, and yet as modern and elegant as possible.
• no backward / forward compatibility needed.
• not copying large fragments around - it is desirable that files are loaded into RAM in the background, and all large chunks are available immediately later.
• should not rely on a goal that has the same consistency and alignment, i.e. a C plugin in Python that dumps its structures to disk would not be a very good idea.
• should allow you to move downloaded data around, since with separate files 1/3 the size of RAM, fragmentation can be a problem. No MMU for abuse.
• Robotism is a great bonus, since my attention span is very short, that is, I would save some of the code and forget the download or vice versa, so at least a dumb guarantee would be good.
• , , .

- Python C, , -/ , /, .

: .

Answer 1

.

, . . .

blob . . , , . , dll-.

ruby, bbq, iphone.

, blob:

// Memory layout
//
// p begining of file in memory.
// p + 0 : num_pointers
// p + 4 : offset 0
// p + 8 : offset 1
// ...
// p + ((num_pointers - 1) * 4) : offset n-1
// p + (num_pointers * 4) : num_pointers   // again so we can figure out 
//                                            what memory to free.
// p + ((num_pointers + 1) * 4) : start of cooked data
//

, blob :

void* bbq_load(const char* filename)
{
    unsigned char* p;
    int size = LoadFileToMemory(filename, &p);
    if(size <= 0)
        return 0;

    // get the start of the pointer table
    unsigned int* ptr_table = (unsigned int*)p;
    unsigned int num_ptrs = *ptr_table;
    ptr_table++;

    // get the start of the actual data
    // the 2 is to skip past both num_pointer values
    unsigned char* base = p + ((num_ptrs + 2) * sizeof(unsigned int));

    // fix up the pointers
    while ((ptr_table + 1) < (unsigned int*)base)
    {
        unsigned int* ptr = (unsigned int*)(base + *ptr_table);
        *ptr = (unsigned int)((unsigned char*)ptr + *ptr);
        ptr_table++;
    }

    return base;
}

My bbq -, , python.

!

Answer 2

, Nintendo GameCube DS, 3D- :

• , , , , .. , , , (Adler-32, CRC-16 ..).
• , 32- .
• .
• endian .
• (), () (gzip). .

: .

- , , - . , , . , .

Answer 3

, " ".: -)

, :

• , , . , , "" ; , , , , , . , , , .

• , - , , , C Python . "" , , , , ( , , , , , .)

Answer 4

Consider storing your data as a BLOB in SQLite DB. SQLite is extremely portable and heavy, ANSI C has both C ++ and Python interfaces. This will take care of large files, lack of fragmentation, variable length records with quick access, etc. The rest is just the serialization of structures for these BLOBs.