Static communication with generated protobuffs causes interruption

Question

I have a project that compiles C ++ generated protobuf serializers into a static library. Executable links to this library and .so (.dll) are also executed. The executable file later downloads the .so file. When this happens, I get:

[libprotobuf ERROR /mf-toolchain/src/protobuf-3.0.0-beta-1/src/google/protobuf/descriptor_database.cc:57] File already exists in database: mri.proto [libprotobuf FATAL /mf-toolchain/src/protobuf-3.0.0-beta-1/src/google/protobuf/descriptor.cc:1128] CHECK failed: generated_database_->Add(encoded_file_descriptor, size): terminate called after throwing an instance of 'google::protobuf::FatalException' what(): CHECK failed: generated_database_->Add(encoded_file_descriptor, size): Aborted (core dumped) 

To be completely clear, I have one static library A, linked to program P and shared library S. Later P loads S, and I get the error above.

I looked at similar errors in stackoverflow and google in general, but I am sure that I only get attached to the library and not recompile the source. As far as I can tell, this should mean that the compiled data is the same.

Also note: this problem only occurs on Linux. This works great on Windows and OS X.

Any suggestions for correcting this are recognized.

Answer 1

The problem is that your static library contains the mri.pb.cc file, which in its global initializers registers type descriptors in the global descriptor database supported by libprotobuf. Since your static library is loaded twice into your program, this initializer works twice, but since you only have one copy of libprotobuf in your process, both initializers register in the same global database and detect a conflict.

To fix this problem, you need to change the static library to a shared library, on which both the main program and the dynamically loaded library depend.

I'm not sure why you see other behavior on Windows or OSX. It is best to assume that on these platforms you are actually linking two separate copies of libprotobuf to your program - one in the main executable file and one in the dynamically loaded library. Thus, there are two copies of the descriptor and conflict database. However, you are likely to see much more subtle issues here. If you ever passed protobuf object pointers between the main program and the dynamically loaded module (without serialization and then parsing), you could have a protobuf object created by one copy of the library, but used with another copy (and therefore another copy of the database descriptor data), which will confuse the library and cause strange things.

Alternatively, if you never pass protobuf objects across the border, you can "fix" the problem on Linux by linking libprotobuf statically to get two copies, as described above. But it is rather dangerous; I would not recommend it.

Answer 2

This error occurred to me in the context of an executable file linking two libraries ( LibA , LibB ), in which both LibB compiled the same proto message, where LibB depends on LibA and links to it.

I was faced with this situation due to the fact that LibA was not previously dependent on any of the proto-buffer infrastructure, and LibB created a complete set of relevant LibB messages for this proprietary tool application to communicate with another application. The new release of LibA required new dependencies on two other libraries that compile various LibC messages ( LibC , LibD ). The problem manifests itself equally in both LibC and LibD , I will discuss LibC because the solution was identical.

At boot time, the application loads LibC , and as a result, it loads the topmost LibB module and that is when the interrupt will be called in LogMessage::Finish() in common.cc . I discovered who was doing this double boot by setting a breakpoint several levels above the interrupt context. Here is the appropriate source to consider for a dual-boot proto message that I call SomeMessage ...

 void LibC_AddDesc_SomeMessage_2eproto() { static bool already_here = false; // <--- Breakpoint Set Here if (already_here) return; already_here = true; 

Hit 1 breakpoint: loading libc

 LibC.dll!MyNamespace::LibC_AddDesc_SomeMessage_2eproto() Line 415 C++ LibC.dll!MyNamespace::LibC_AddDesc_ParentMessage_2eproto() Line 388 C++ LibC.dll!MyNamespace::StaticDescriptorInitializer_ParentMessage_2eproto::StaticDescriptorInitializer_ParentMessage_2eproto() Line 494 C++ LibC.dll!MyNamespace::'dynamic initializer for 'static_descriptor_initializer_ParentMessage_2eproto_''() Line 495 + 0x21 bytes C++ msvcr100d.dll!_initterm() + 0x2c bytes LibC.dll!_CRT_INIT(void * hDllHandle, unsigned long dwReason, void * lpreserved) Line 289 C LibC.dll!__DllMainCRTStartup(void * hDllHandle, unsigned long dwReason, void * lpreserved) Line 506 + 0x13 bytes C LibC.dll!_DllMainCRTStartup(void * hDllHandle, unsigned long dwReason, void * lpreserved) Line 477 C ntdll.dll!LdrpRunInitializeRoutines() + 0x1e8 bytes ntdll.dll!LdrpInitializeProcess() - 0x14c9 bytes ntdll.dll!string "Enabling heap debug options\n"() + 0x29a99 bytes ntdll.dll!LdrInitializeThunk() + 0xe bytes 

I could see during LibC loading that the breakpoint was hit twice, and the static variable already_here was set from false to true, and held true and skipped logging this message.

Hit 2 breakpoint: loading libb

When this library tries to load, the already_here variable will already_here be initialized to false, and we will continue to try to register this message a second time, which caused an interrupt.

 LibB.dll!MyNamespace::LibC_AddDesc_SomeMessage_2eproto() Line 415 C++ LibB.dll!MyNamespace::LibC_AddDesc_ParentMessage_2eproto() Line 388 C++ LibB.dll!MyNamespace::LibC_AddDesc_FullMessage_2eproto() Line 219 C++ LibB.dll!MyNamespace::StaticDescriptorInitializer_FullMessage_2eproto::StaticDescriptorInitializer_FullMessage_2eproto() Line 358 C++ LibB.dll!MyNamespace::'dynamic initializer for 'static_descriptor_initializer_FullMessage_2eproto_''() Line 359 + 0x21 bytes C++ msvcr100d.dll!_initterm() + 0x2c bytes LibB.dll!_CRT_INIT(void * hDllHandle, unsigned long dwReason, void * lpreserved) Line 289 C LibB.dll!__DllMainCRTStartup(void * hDllHandle, unsigned long dwReason, void * lpreserved) Line 506 + 0x13 bytes C LibB.dll!_DllMainCRTStartup(void * hDllHandle, unsigned long dwReason, void * lpreserved) Line 477 C ntdll.dll!LdrpRunInitializeRoutines() + 0x1e8 bytes ntdll.dll!LdrpInitializeProcess() - 0x14c9 bytes ntdll.dll!string "Enabling heap debug options\n"() + 0x29a99 bytes ntdll.dll!LdrInitializeThunk() + 0xe bytes 

... and we get to stubs / common.cc on the interrupt line

 void LogMessage::Finish() { bool suppress = false; if (level_ != LOGLEVEL_FATAL) { InitLogSilencerCountOnce(); MutexLock lock(log_silencer_count_mutex_); suppress = internal::log_silencer_count_ > 0; } if (!suppress) { internal::log_handler_(level_, filename_, line_, message_); } if (level_ == LOGLEVEL_FATAL) { abort(); // <----- runtime crash! } } 

And in std :: err you will find the following text ...

 libprotobuf ERROR descriptor_database.cc:57] File already exists in database: SomeMessage.proto libprotobuf FATAL descriptor.cc:860] CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 

The solution was simple, I opened the LibC project, LibC for pb and deleted these proto messages from LibB . The same thing was done for LibD .