I watched DynObj and decided to do my own experimenting with vftables. I work with Visual Studio 2010 and created a main console that creates an object with a virtual function that returns std :: string.
The test code basically tries to call the public virtual function of the object using a pointer obtained from the vftable object. I found that there is no problem for a function returning a primitive type. But when the function returns std :: string, the compiler inserts the effective pop stack (add esp, 4). This causes a stack validation code, which is followed by an exception.
I noticed that this is the norm for functions declared in the global space. But the function inside the class does not generate an ESP modifier mail call.
Here's the gist of the code along with the assembly ...
class VClass
{
public:
VClass() {}
~VClass() {}
virtual std::string GetString() {return "vstring";}
};
std::string StrFunc()
{
return "string";
}
void main(int argc, char* argv[])
{
VClass vClass;
__int32 ** vftabletable;
vftabletable = (__int32**)&vClass;
StrFunc();
vClass.GetString();
((std::string (*)())vftabletable[0][0])();
}
The last call to the virtual function through the cast pointer results in:
Runtime Check Error # 0 - The ESP value was not properly stored on a function call. Usually this is the result of calling a function declared using one calling convention with a function pointer declared with another conditional call.
So my question is: how do I invoke the correct calling convention for a virtual function that returns a non-primitive type?
Btw, when I split into 00BE178B and set the next statement to 00BE178E, execution completes without problems.