The size of the pointer will not change; which depends on the platform and modularity, and not on the compiler (32-bit and 64-bit, etc.).
What may differ from the size of basically everything else, and what will differ from the templates.
Filling and aligning structures usually depends on the compiler and often depends on the parameters in the compiler. There are such free rules as pointers, usually located on the platform-bit boundary, and bools with 3 bytes after them, but before the compiler, how to handle this.
Templates, especially from STL (which differ for each compiler), can have different elements, sizes, additions and basically anything. The only standard part is the API, the backend remains in the STL implementation (there are some rules, but compilers can still compile the templates). Passing templates between modules from the same assembly is bad enough, but between different compilers it can often be fatal.
Things that are not standardized (name changes) or are especially specific if necessary (memory allocation) will also be incompatible. You can work around both of these problems by destroying only the library that creates (in any case, good practice) and using the STL objects that accept the debit to distribute and export using non-address names and / or C style ( extern "C" ) for exported methods.
It also seems to me that you remember how compilers handle virtual destructors in a vtable with a slight difference.
If you can only make references to your own objects, completely avoid externally visible templates, primarily work with pointers and exported or virtual methods, you can avoid the vast majority of problems (COM does just that, for compatibility with most compilers and languages). It may be a pain to write, but if you need this compatibility, it is possible.
To alleviate some problems, but not all, using an alternative to STL (such as the Qt core library) will fix this particular problem. While throwing Qt into any old project is disgusting waste and will cause more thought than "reinforce ALL THINGS !!!" philosophy, this may be useful for decoupling the library and the compiler to a greater extent than using the STL stock file.