First, ByVal .. As Any for the _Out_ argument _Out_ not a good idea (I'm not even sure if this is possible); if you use ByVal for this, you want it to be As Long (see below why).
So, for APIs with one or more _Out_ arguments intended to represent the location of the buffer / variable / memory, there are two ways (for each corresponding argument in any case) to write a declaration, depending on what you want to pass:
ByRef lpBuffer As Any or just lpBuffer As Any : you use this in the declaration for the _Out_ argument if you intend to pass a valid variable when calling the API where the data should be copied to. For example, you can use an array of bytes as follows:
Private Declare Function ReadProcessMemory Lib "kernel32" (ByVal hProcess As Long, _ ByVal lpBaseAddress As Long, lpBuffer As Any, ByVal nSize As Long, _ lpNumberOfBytesWritten As Long) As Long '[..] Dim bytBuffer(255) As Byte, lWrittenBytes As Long, lReturn As Long lReturn = ReadProcessMemory(hTargetProcess, &H400000&, bytBuffer(0), 256, lWrittenBytes)
Note that the called (here, ReadProcessMemory() ) fills everything you provide as lpBuffer with data, regardless of the actual size of the passed variable. Therefore, the size of the buffer must be provided through nSize , because otherwise the called user will not be able to find out the size of the provided buffer. Also note that we are passing the first element of the (byte) array, since that is where the caller should begin to write data.
With the same declaration, you can even go long if you want (if, for example, what you want to get is an address or DWord value of some kind), but then nSize should be 4 bytes (at most).
Also note that the last argument lpNumberOfBytesWritten also an _Out_ argument and passed to ByRef, but you do not need to give the caller its size; that since there is an agreement between the caller and the callee that any variable is passed, exactly 4 bytes will always be written to it.
ByVal lpBuffer As Long : you use this in the declaration for the _Out_ argument if, when calling the API, you intend to pass the memory cell as a 32-bit value (i.e. a pointer); the passed Long value will not be changed, what will be overwritten is the memory cell referenced by the Long value. Repeating the same example, but with a slightly different declaration, we get:
Private Declare Function ReadProcessMemory Lib "kernel32" (ByVal hProcess As Long, _ ByVal lpBaseAddress As Long, ByVal lpBuffer As Long, ByVal nSize As Long, _ lpNumberOfBytesWritten As Long) As Long '[..] Dim bytBuffer(255) As Byte, lPointer As Long, lWrittenBytes As Long, lReturn As Long lPointer = VarPtr(bytBuffer(0)) lReturn = ReadProcessMemory(hTargetProcess, &H400000&, lPointer, 256, lWrittenBytes) ' If we want to make sure the value of lPointer didn't change: Debug.Assert (lPointer = VarPtr(bytBuffer(0)))
See, this is practically the same again, with the only difference being that we provide a pointer (memory address) to bytBuffer , rather than passing bytBuffer directly. We could even provide the value returned by VarPtr() directly, instead of using Long (here lPointer ):
lReturn = ReadProcessMemory(hTargetProcess, &H400000&, VarPtr(bytBuffer(0)), 256, _ lWrittenBytes)
Warning No. 1 . For _Out_ arguments, if you declare them ByVal , they should always be As Long . This is because the calling convention expects the value to consist of exactly 4 bytes (32-bit value / DWORD). For example, if you have to pass a value using the Integer type, you will get unexpected behavior, because what will be used as the value for the memory cell is 2 bytes of this Integer plus the next 2 bytes that arrive immediately after the contents of this Integer variable in memory that could be anything. And if that happens, this is the place of memory that the challenge will write to, you are likely to crash.
Warning # 2 : you DO NOT want to use VarPtrArray() (which must be explicitly declared anyway), since the return value will be the address of the SAFEARRAY structure of the array (number of elements, element size, etc.), and not a pointer to the array data (which is the same address as the first element in the array).
In essence, for Win32 arguments, the API (i.e. stdcall) is always always passed as 32-bit values . The value of these 32-bit values will depend on the expected specific API, so its declaration should reflect this. So:
- whenever a
ByRef argument is declared, the memory location of any variable passed in will be used - whenever a
ByVal .. As Long argument is ByVal .. As Long , the (32-bit) value of any variable will be used (the value does not have to be a memory cell, for example, the hProcess ReadProcessMemory() argument).
Finally, even if you declare the _Out_ ByRef argument (or, for example, the way the API is declared, and you cannot change it, because if it comes from typelib), you can always pass a pointer from the actual variable by adding ByVal before as when making a call . Returning to the first declaration of ReadProcessMemory() (when lpBuffer declared ByRef ), we will do the following:
Private Declare Function ReadProcessMemory Lib "kernel32" (ByVal hProcess As Long, _ ByVal lpBaseAddress As Long, lpBuffer As Any, ByVal nSize As Long, _ lpNumberOfBytesWritten As Long) As Long '[..] Dim bytBuffer(255) As Byte, lWrittenBytes As Long, lReturn As Long lReturn = ReadProcessMemory(hTargetProcess, &H400000&, ByVal VarPtr(bytBuffer(0)), 256, _ lWrittenBytes)
Adding ByVal tells the compiler that what should be passed on the stack is not the VarPtr() address, but instead the value returned by VarPtr(bytBuffer(0)) . But if the argument was declared ByVal .. As Long , then you have no choice, you can pass a pointer (i.e. the address of the memory cell).
NOTA: this answer, accepted throughout the architecture under discussion, was IA32 or emulate it