Load a custom DLL programmatically from another path

[yinkajewole]

I have a DLL with I don't want to put it in the system folder.
I do I let VBA, how do I load dll from the path I want?

 

[Tragic Shadow]

The oleaut32 library exports this little used but extremely useful api function called DispCallFunc... With this function, one can call other functions (apis or otherwise) which have more than 4 arguments.

Here is a generic helper function DllStdCall that wraps the above ole api:

Option Explicit

#If VBA7 Then
    Private Declare PtrSafe Function DispCallFunc Lib "oleaut32.dll" (ByVal pvInstance As LongPtr, ByVal FuncAddr As LongPtr, ByVal CallConvention As Integer, ByVal rtnType As VbVarType, ByVal FuncArgsCnt As Long, ByRef FuncArgTypes As Any, ByRef FuncArgVarAddresses As Any, ByRef FuncResult As Any) As Long
    Private Declare PtrSafe Sub SetLastError Lib "kernel32.dll" (ByVal dwErrCode As Long)
    Private Declare PtrSafe Function MessageBoxExW Lib "user32" (ByVal hwnd As LongPtr, ByVal lpText As LongPtr, ByVal lpCaption As LongPtr, ByVal uType As Long, ByVal wLanguageId As Long) As Long
    Private Declare PtrSafe Function GetProcAddress Lib "kernel32" (ByVal hModule As LongPtr, ByVal lpProcName As String) As LongPtr
    Private Declare PtrSafe Function LoadLibrary Lib "kernel32" Alias "LoadLibraryA" (ByVal lpLibFileName As String) As LongPtr
    Private Declare PtrSafe Function FreeLibrary Lib "kernel32" (ByVal hLibModule As LongPtr) As Long
#Else
    Private Declare Function DispCallFunc Lib "oleaut32.dll" (ByVal pvInstance As Long, ByVal FuncAddr As Long, ByVal CallConvention As Integer, ByVal rtnType As VbVarType, ByVal FuncArgsCnt As Long, ByRef FuncArgTypes As Any, ByRef FuncArgVarAddresses As Any, ByRef FuncResult As Any) As Long
    Private Declare Sub SetLastError Lib "kernel32.dll" (ByVal dwErrCode As Long)
    Private Declare Function MessageBoxExW Lib "user32" (ByVal hwnd As Long, ByVal lpText As Long, ByVal lpCaption As Long, ByVal uType As Long, ByVal wLanguageId As Long) As Long
    Private Declare Function GetProcAddress Lib "kernel32" (ByVal hModule As Long, ByVal lpProcName As String) As Long
    Private Declare Function LoadLibrary Lib "kernel32" Alias "LoadLibraryA" (ByVal lpLibFileName As String) As Long
    Private Declare Function FreeLibrary Lib "kernel32" (ByVal hLibModule As Long) As Long
#End If

#If Win64 Then
    Private Function DllStdCall(ByVal pAddr As LongLong, ByVal FunctionReturnType As Long, ParamArray FunctionParameters() As Variant) As Variant
 
        Dim vParamPtr() As LongLong
#Else
    Private Function DllStdCall(ByVal pAddr As Long, ByVal FunctionReturnType As Long, ParamArray FunctionParameters() As Variant) As Variant
 
        Dim vParamPtr() As Long
#End If

    Const CC_STDCALL    As Long = 4
 
    If Not (FunctionReturnType And &HFFFF0000) = 0& Then Exit Function

    Dim pIndex As Long, pCount As Long
    Dim vParamType() As Integer
    Dim vRtn As Variant, vParams() As Variant

    vParams() = FunctionParameters()
    pCount = Abs(UBound(vParams) - LBound(vParams) + 1&)
    If pCount = 0& Then
        ReDim vParamPtr(0 To 0)
        ReDim vParamType(0 To 0)
    Else
        ReDim vParamPtr(0 To pCount - 1&)
        ReDim vParamType(0 To pCount - 1&)
        For pIndex = 0& To pCount - 1&
            vParamPtr(pIndex) = VarPtr(vParams(pIndex))
            vParamType(pIndex) = VarType(vParams(pIndex))
        Next
    End If
 
    pIndex = DispCallFunc(ByVal 0&, pAddr, CC_STDCALL, FunctionReturnType, pCount, vParamType(0), vParamPtr(0), vRtn)
    If pIndex = 0& Then
        DllStdCall = vRtn
    Else
        SetLastError pIndex
    End If

End Function

And here is an example that shows how to use the DllStdCall wrapper in order to dynamically call an api function with more than 4 arguments.

In this test, I have chosen the MessageBoxExW api which is exported by the User32 Windows library and which has 5 arguments but you can choose any other api function in the same fashion.

Sub Test()

    #If Win64 Then
        Dim hLib As LongLong, hProcAddr As LongLong
    #Else
        Dim hLib As Long, hProcAddr As Long
    #End If
  
    Const MB_ICONINFORMATION = &H40&
  
    hLib = LoadLibrary("user32")
    If hLib Then
        hProcAddr = GetProcAddress(hLib, "MessageBoxExW")
        If hProcAddr Then
            DllStdCall hProcAddr, vbLong, Application.hwnd, StrPtr("Hello World!"), StrPtr("Tile"), MB_ICONINFORMATION, 0
        End If
    End If
    Call FreeLibrary(hLib)

End Sub

EDIT:
You need to make sure you pass the correct expected types for the arguments and for the function return , failing to do so can crash the application.

Jaafar, I really appreciate your help.
I was stuck with EnumEnabledLayoutOrTip function from input.dll. You just saved a lot of my hairs.
I got the list of keyboard layouts from registry already but with the help of your code on that enum function, it is way much simpler and easier now.

But there remained another hurdle: GetLayoutDescription from input.dll.
Found info from: https://www.cyberforum.ru/post15119827.html, according to it, the function can be called as:
HRESULT GetLayoutDescription (LPWSTR szId, LPCWSTR pszName, LPUINT uBufLength, DWORD dwFlags)
MS site showed no reference on this one or another similar function, GetDefaultLayout. They both produced the same error.
To call it, I tried:
1.The Trick's patchfunc but I think it's just usable before AddressOf was introduced and VBA6.dll is a pain to find on the web and even after I found it, I can't get it working.
2.CallWindowProc function cannot be used. It just crashes Excel.
3.DllStdCall returned error, Err.LastDllError is -2147352560 Invalid Callee.

This function is exported for sure. I checked in Dll Export Viewer, PE Studio, PE Explorer, Ollydbg.
But I can't call it. The GetProcAddress is returning fine. I tried switching strptr around and LastDllError returned "wrong argument type" only.
So, I think the main problem is Invalid Callee not the wrong variable types.
I thought Exported functions were callable using any of the above methods.

If EnumEnabledLayoutOrTip can be called with DllStdCall, why does GetLayoutDescription crash every time it is called?
Win10 64bit with Excel2010 32bit.
May be I am required to start a new thread now. If so, please let me know. I have no intention of hijacking this thread.

 

[Tragic Shadow]

Jaafar, I really appreciate your help.
I was stuck with EnumEnabledLayoutOrTip function from input.dll. You just saved a lot of my hairs.
I got the list of keyboard layouts from registry already but with the help of your code on that enum function, it is way much simpler and easier now.

But there remained another hurdle: GetLayoutDescription from input.dll.
Found info from: https://www.cyberforum.ru/post15119827.html, according to it, the function can be called as:
HRESULT GetLayoutDescription (LPWSTR szId, LPCWSTR pszName, LPUINT uBufLength, DWORD dwFlags)
MS site showed no reference on this one or another similar function, GetDefaultLayout. They both produced the same error.
To call it, I tried:
1.The Trick's patchfunc but I think it's just usable before AddressOf was introduced and VBA6.dll is a pain to find on the web and even after I found it, I can't get it working.
2.CallWindowProc function cannot be used. It just crashes Excel.
3.DllStdCall returned error, Err.LastDllError is -2147352560 Invalid Callee.

This function is exported for sure. I checked in Dll Export Viewer, PE Studio, PE Explorer, Ollydbg.
But I can't call it. The GetProcAddress is returning fine. I tried switching strptr around and LastDllError returned "wrong argument type" only.
So, I think the main problem is Invalid Callee not the wrong variable types.
I thought Exported functions were callable using any of the above methods.

If EnumEnabledLayoutOrTip can be called with DllStdCall, why does GetLayoutDescription crash every time it is called?
Win10 64bit with Excel2010 32bit.
May be I am required to start a new thread now. If so, please let me know. I have no intention of hijacking this thread.

OMG, I got it working just now. It's just like Jaafar said, the type of the argument was wrong.
It's my fault. Returning the expected value now but pIndex is still Invalid Callee.
Nevertheless, it is working now.

 

[Jaafar Tribak]

HRESULT GetLayoutDescription (LPWSTR szId, LPCWSTR pszName, LPUINT uBufLength, DWORD dwFlags)

According to the documentation:
LPWSTR and LPCWSTR are pointers to null-terminated unicode character arrays
LPUINT, DWORD and HRESULT are Longs.

So, I guess, your DllStdCall should look something like this :
DllStdCall hProcAddr, vbLong, StrPtr(szId), StrPtr(pszName), uBufLength, dwFlags

Getting the argument variable types as well as the function return type right is indispensable.

Note that API functions with strings come into two flavors : ANSI and UNICODE.
So, if an api function expects an ANSI string, we need to first convert our VBA string (ie: a BSTR which in fact, is a pointer to a null-terminated Unicode character array that is preceeded by a 4-byte length field) to an ANSI string.

In my previous post, I gave the example of the UNICODE version of the MessageBoxExW API.

If we were to use the ANSI version (ie: MessageBoxExA) , we would do something like this :

Method 1 : Call with DECLARE statement:

Option Explicit

#If VBA7 Then
    Declare PtrSafe Function MessageBoxExA Lib "user32" _
                (ByVal hwnd As LongPtr, _
                ByVal lpText As LongPtr, _
                ByVal lpCaption As LongPtr, _
                ByVal uType As Long, _
                ByVal wLanguageId As Long _
            ) As Long
#Else
    Declare Function MessageBoxExA Lib "user32" _
                (ByVal hwnd As Long, _
                ByVal lpText As Long, _
                ByVal lpCaption As Long, _
                ByVal uType As Long, _
                ByVal wLanguageId As Long _
            ) As Long
#End If

Sub Test1()

    'ANSI vaersion of the MessageBoxEx api.

    Const MB_ICONINFORMATION = &H40&
    Dim sPrompt As String, sTitle As String
    Dim ar1() As Byte, ar2() As Byte

    sPrompt = "Hello World!"
    sTitle = "Title"
 
    ar1 = VBA.StrConv(sPrompt, vbFromUnicode)
    ar2 = VBA.StrConv(sTitle, vbFromUnicode)
 
    MessageBoxExA Application.hwnd, StrPtr(ar1), StrPtr(ar2), MB_ICONINFORMATION, 0

End Sub

Method 2 : MessageBoxA api loaded dynamically: (DllStdCall approach)

Option Explicit

#If VBA7 Then
    Private Declare PtrSafe Function DispCallFunc Lib "oleaut32.dll" (ByVal pvInstance As LongPtr, ByVal FuncAddr As LongPtr, ByVal CallConvention As Integer, ByVal rtnType As VbVarType, ByVal FuncArgsCnt As Long, ByRef FuncArgTypes As Any, ByRef FuncArgVarAddresses As Any, ByRef FuncResult As Any) As Long
    Private Declare PtrSafe Sub SetLastError Lib "kernel32.dll" (ByVal dwErrCode As Long)
    Private Declare PtrSafe Function GetProcAddress Lib "kernel32" (ByVal hModule As LongPtr, ByVal lpProcName As String) As LongPtr
    Private Declare PtrSafe Function LoadLibrary Lib "kernel32" Alias "LoadLibraryA" (ByVal lpLibFileName As String) As LongPtr
    Private Declare PtrSafe Function FreeLibrary Lib "kernel32" (ByVal hLibModule As LongPtr) As Long
#Else
    Private Declare Function DispCallFunc Lib "oleaut32.dll" (ByVal pvInstance As Long, ByVal FuncAddr As Long, ByVal CallConvention As Integer, ByVal rtnType As VbVarType, ByVal FuncArgsCnt As Long, ByRef FuncArgTypes As Any, ByRef FuncArgVarAddresses As Any, ByRef FuncResult As Any) As Long
    Private Declare Sub SetLastError Lib "kernel32.dll" (ByVal dwErrCode As Long)
    Private Declare Function GetProcAddress Lib "kernel32" (ByVal hModule As Long, ByVal lpProcName As String) As Long
    Private Declare Function LoadLibrary Lib "kernel32" Alias "LoadLibraryA" (ByVal lpLibFileName As String) As Long
    Private Declare Function FreeLibrary Lib "kernel32" (ByVal hLibModule As Long) As Long
#End If

Sub Test2()

    'ANSI vaersion of the MessageBoxEx api.
    #If Win64 Then
        Dim hLib As LongLong, hProcAddr As LongLong
    #Else
        Dim hLib As Long, hProcAddr As Long
    #End If
 
    Const MB_ICONINFORMATION = &H40&
    Dim sPrompt As String, sTitle As String
    Dim ar1() As Byte, ar2() As Byte

    sPrompt = "Hello World!"
    sTitle = "Title"
 
    ar1 = VBA.StrConv(sPrompt, vbFromUnicode)
    ar2 = VBA.StrConv(sTitle, vbFromUnicode)
 
    hLib = LoadLibrary("user32")
    If hLib Then
        hProcAddr = GetProcAddress(hLib, "MessageBoxExA")
        If hProcAddr Then
            DllStdCall hProcAddr, vbLong, Application.hwnd, StrPtr(ar1), StrPtr(ar2), MB_ICONINFORMATION, 0
        End If
    End If
    Call FreeLibrary(hLib)

End Sub

#If Win64 Then
    Private Function DllStdCall(ByVal pAddr As LongLong, ByVal FunctionReturnType As Long, ParamArray FunctionParameters() As Variant) As Variant
 
        Dim vParamPtr() As LongLong
#Else
    Private Function DllStdCall(ByVal pAddr As Long, ByVal FunctionReturnType As Long, ParamArray FunctionParameters() As Variant) As Variant
 
        Dim vParamPtr() As Long
#End If

    Const CC_STDCALL    As Long = 4
 
    If Not (FunctionReturnType And &HFFFF0000) = 0& Then Exit Function

    Dim pIndex As Long, pCount As Long
    Dim vParamType() As Integer
    Dim vRtn As Variant, vParams() As Variant

    vParams() = FunctionParameters()
    pCount = Abs(UBound(vParams) - LBound(vParams) + 1&)
    If pCount = 0& Then
        ReDim vParamPtr(0 To 0)
        ReDim vParamType(0 To 0)
    Else
        ReDim vParamPtr(0 To pCount - 1&)
        ReDim vParamType(0 To pCount - 1&)
        For pIndex = 0& To pCount - 1&
            vParamPtr(pIndex) = VarPtr(vParams(pIndex))
            vParamType(pIndex) = VarType(vParams(pIndex))
        Next
    End If
 
    pIndex = DispCallFunc(ByVal 0&, pAddr, CC_STDCALL, FunctionReturnType, pCount, vParamType(0), vParamPtr(0), vRtn)
    If pIndex = 0& Then
        DllStdCall = vRtn
    Else
        SetLastError pIndex
    End If

End Function

It's my fault. Returning the expected value now but pIndex is still Invalid Callee.

Not sure why you still get that Invalid Callee error code.

 

[Tragic Shadow]

Jaafar, Sir, I found out the reason behind Invalid Callee now.
I found that after each call to the DllStdCall, the vRtn is actually 0 and the function did get me the LayoutDescription so I thought that the function at the address is working fine.
The problem must be with DispCallFunc. Well, not a problem but more like a warning.
I read around and found that the CC_STDCALL must be the real issue here.
So, I switched to CC_CDECL As Long = 1 et voila! pIndex=0 S_OK.
I think this is because this function is an undocumented one and it must be written in C or C++.

I am not sure that the source documentation is very right about the argument types because I found that if I call like:

DllCDECLCall hProcAddr, vbLong, StrPtr(szId), StrPtr(pszName), uBufLength, dwFlags

Excel crashes. So I had to change to:

1. StrPtr(szId) to szId (as a string, not a pointer)
2. StrPtr(pszName) as it is (it can't be changed to pszName and if it were, crashes again)
3. uBufLength has to be just an Integer, if it were declared as a Long, it crashes, so is passing a value like 256
4. dwFlags is a Long yes but it has to be 0 or else it crashes.

Anyway, I really appreciate your kind help. You have motiviated me to go on, otherwise, I'd have happily stopped solving that Invalid Callee as soon as the function do what I want.

Now, only the last hurdle remains: GetDefaultLayout, of which I have no idea what the input and output arguments are.
I don't think there's any method or tool to find out a function's input and output arguments rather than testing blindly and crashing a thousand and one times.
I found that Ollydbg has a feature to loadup a DLL and pass arugments to exported functions but I will have to learn about it all over again.
When I finished everything, I will come back here and share a link to my GitHub where I will explain everything.
Thanks again, Jaafar, you rock!

 

[Jaafar Tribak]

The problem must be with DispCallFunc. Well, not a problem but more like a warning.
I read around and found that the CC_STDCALL must be the real issue here.
So, I switched to CC_CDECL As Long = 1 et voila! pIndex=0 S_OK.

Good catch. I forgot about mentioning the calling convention.

I don't think there's any method or tool to find out a function's input and output arguments

I no longer have Ollydbg but if I remember correctly, it did show the arguments var types, so does Visual Studio.

Let us know when you figure this out and thanks for the feedback.