The fastest way to read bytes in D2

Question

I want to read single bytes from a file into a D2 application as quickly as possible. An application needs byte per byte, so reading large blocks of data is not an option for the interface for the reader.

For this, I created some trivial implementations in C ++, Java, D2 at: https://github.com/gizmomogwai/performance .

As you can see, I tried simple reads, buffers in the application code, and memory mapped files. For my utility, a memory mapping solution worked best, but it is strange that D2 is slower than java. I would hope that D2 will land between C ++ and Java (C ++ code compiled with -O3 -g, D2 code compiled with -O -release).

So, please tell me what I'm doing wrong here, and how to speed up the implementation of D2.

To give you an idea of ​​the use case, here is a C ++ implementation:

class StdioFileReader { private: FILE* fFile; static const size_t BUFFER_SIZE = 1024; unsigned char fBuffer[BUFFER_SIZE]; unsigned char* fBufferPtr; unsigned char* fBufferEnd; public: StdioFileReader(std::string s) : fFile(fopen(s.c_str(), "rb")), fBufferPtr(fBuffer), fBufferEnd(fBuffer) { assert(fFile); } ~StdioFileReader() { fclose(fFile); } int read() { bool finished = fBufferPtr == fBufferEnd; if (finished) { finished = fillBuffer(); if (finished) { return -1; } } return *fBufferPtr++; } private: bool fillBuffer() { size_t l = fread(fBuffer, 1, BUFFER_SIZE, fFile); fBufferPtr = fBuffer; fBufferEnd = fBufferPtr+l; return l == 0; } }; size_t readBytes() { size_t res = 0; for (int i=0; i<10; i++) { StdioFileReader r("/tmp/shop_with_ids.pb"); int read = r.read(); while (read != -1) { ++res; read = r.read(); } } return res; } 

which is much faster compared to the "same" solution in D:

 struct FileReader { private FILE* fFile; private static const BUFFER_SIZE = 8192; private ubyte fBuffer[BUFFER_SIZE]; private ubyte* fBufferPtr; private ubyte* fBufferEnd; public this(string fn) { fFile = std.c.stdio.fopen("/tmp/shop_with_ids.pb", "rb"); fBufferPtr = fBuffer.ptr; fBufferEnd = fBuffer.ptr; } public int read(ubyte* targetBuffer) { auto finished = fBufferPtr == fBufferEnd; if (finished) { finished = fillBuffer(); if (finished) { return 0; } } *targetBuffer = *fBufferPtr++; return 1; } private bool fillBuffer() { fBufferPtr = fBuffer.ptr; auto l = std.c.stdio.fread(fBufferPtr, 1, BUFFER_SIZE, fFile); fBufferEnd = fBufferPtr + l; return l == 0; } } size_t readBytes() { size_t count = 0; for (int i=0; i<10; i++) { auto reader = FileReader("/tmp/shop_with_ids.pb"); ubyte buffer[1]; ubyte* p = buffer.ptr; auto c = reader.read(p); while (1 == c) { ++count; c = reader.read(p); } } return count; } 

Answer 1

This is very likely due to sfread . No one guarantees that it will do the same in D as in C - you most likely use a completely different CRT (unless you use the Digital Mars C ++ compiler?).

This means that the library can do things like synchronization, etc., which slow down the work. The only way you can know is to force D to use the same library as C by telling the linker the link to the same libraries.

Until you can do this, you are comparing apples to oranges. If this is not possible, then call the OS directly from both and then compare the results - this way you are guaranteed that the main call will be the same for both.

Answer 2

what happens if you use the std.stdio module:

 import std.stdio; struct FileReader { private File fFile; private enum BUFFER_SIZE = 8192;//why not enum? private ubyte[BUFFER_SIZE] fBuffer=void;//avoid (costly) initialization to 0 private ubyte[] buff; public this(string fn) { fFile = File("/tmp/shop_with_ids.pb", "rb"); } /+ public ~this(){//you really should have been doing this if you used std.c.stdio.fopen //but it unnecessary for std.stdio File (it ref counted) fFile.close(); } +/ public int read(out ubyte targetBuffer) { auto finished = buff.length==0; if (finished) { finished = fillBuffer(); if (finished) { return 0; } } targetBuffer = buff[0]; buff = buff[1..$]; return 1; } private bool fillBuffer() { if(!fFile.isOpen())return false; buff = fFile.rawRead(fBuffer[]); return buff.length>0; } } size_t readBytes() { size_t count = 0; for (int i=0; i<10; i++) { auto reader = FileReader("/tmp/shop_with_ids.pb"); ubyte buffer; auto c = reader.read(buffer); while (1 == c) { ++count; c = reader.read(buffer); } } return count; } 

if you want to compare the true speed that you have to compile with -release -O -inline (this disables debugging (mainly OOB array checks), optimizes and embeds what it can) (and, of course, is similar to solution C ++)