Windows Support

[mon]

It would be lovely if I could use this library on Windows, as my driver only comes in dll form.

I'm comfortable with the WinAPI and C, but cython is doing my head in.

Would a potential "easy" fix be using ctypes.dll.LoadLibrary, since that's cross-platform? It seems like there might be some boilerplate behind the scenes to cast function types.

From what I can tell, the current method loads the symbols globally, which makes it "just work". I can't see any reference to the dll itself past the load method.

[danni]

Hi, yes, at the moment it's globally linked, and was easier from the POV of Cython; and while it does make things like supporting multiple drivers more difficult, there's already other multiplexers in existence, in P11Kit, etc. I did consider just using P11Kit's loader, which would remove all need for custom dlopen code, but it adds another dependency into a thing that people are going to try and run on vintage enterprise Linux.

You're right it loads the functions into the global symbol space. I don't know if there an equivalent for LoadLibrary, we could completely refactor it to use a handle, or we could port to using the P11Kit loader.

ctypes is a different beast, so that's not going to work, but you could simply call LoadLibrary directly. There's a good chance it has a built in wrapper as dlopen does, otherwise it's easy enough to write one.

[danni]

dlfcn-win32 supports RTLD_GLOBAL and probably could help avoid rewriting _loader altogether.

Otherwise what I think we should do is rewrite loader to produce a struct with all of the functions we need iterated onto it, which could easily use either dlopen/dlsym or LoadLibrary/GetProcAddress (and perhaps support multiple drivers in the future).

[mon]

I think a full rewrite may be unnecessary, as dlfcn-win32 looks fully featured enough to be a dropin replacement (I didn't know it existed). Making a struct of functions seems closer to re-implementing P11Kit anyway.

[danni]

Are you comfortable enough to give it a go for the Windows Cython build? I've not tried to build Cython on Windows before and I don't have a Windows development env to try it on.

[mon]

The build itself "works", just missing dlfcn.h ;) I'll see what I can do - might get some ugly pxds out of it, but it should work.

[mon]

Turns out, no dice. I got about as far as my own (separate) pure C implementation, only to be hit by the linker wondering where all the functions were actually implemented. I got around that in pure C by generating a .lib for my driver dll, but that kills cross compatibility, as expected.

It seems the only way to be crossplatform is to use some sort of LoadLibrary call or similar.

For the moment I'll move back to PyKCS11 - it has a bug that is preventing me from upgrading to Python 3, but 2 isn't so bad :) Thanks for the assistance!

I'll close this, since "fixing" the import situation is more trouble than it's worth.

[danni]

I'd like to keep it open, since it's a big missing part, and this is a useful record of what we've tried so far. The first thing I would try is linker flags to delay resolving the symbols. Then we can change to accessing the symbols through a handle.

For reference: https://joscor.com/blog/load-pkcs11-library-dll-in-c-c/

[mon]

Ok, sounds good! It may be worth noting that the PKCS library has its own function pointer table (at least it seemed that way) which might make assembling all the functions simpler.

[luisza]

You can create a build package for windows using dlfcn-win32, without need to include as dependency with something like python setup.py bdist_wininst. Windows system is required for create a build package, but can solve the windows distribution.

[luisza]

Hi, I continue trying to support windows, so I include dlfcn-win32 and try to compile and works to build _loader.pyx, but I have other problem building 'pkcs11._pkcs11' extension

My changes are here master...luisza:winbuild

building 'pkcs11._pkcs11' extension
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DCK_PTR=* "-DCK_DEFINE_FUNCTION(returnType, name)=returnType name" "-DCK_DECLARE_FUNCTION(returnType, name)=returnType name" "-DCK_DECLARE_FUNCTION_POINTER(returnType, name)=returnType (* name)" "-DCK_CALLBACK_FUNCTION(returnType, name)=returnType (* name)" -IC:\Users\usuario\AppData\Local\Programs\Python\Python36-32\include -IC:\Users\usuario\AppData\Local\Programs\Python\Python36-32\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\8.1\include\shared" "-IC:\Program Files (x86)\Windows Kits\8.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\winrt" /Tcpkcs11\_pkcs11.c /Fobuild\temp.win32-3.6\Release\pkcs11\_pkcs11.obj
_pkcs11.c
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1163): error C2081: 'CK_NOTIFY': name in formal parameter list illegal
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1167): error C2091: function returns function
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1184): error C2059: syntax error: '<parameter-list>'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1192): error C2059: syntax error: '<parameter-list>'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1198): error C2059: syntax error: '<parameter-list>'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1206): error C2059: syntax error: '<parameter-list>'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1215): error C2061: syntax error: identifier 'CK_CREATEMUTEX'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1216): error C2061: syntax error: identifier 'DestroyMutex'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1216): error C2059: syntax error: ';'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1217): error C2061: syntax error: identifier 'LockMutex'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1217): error C2059: syntax error: ';'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1218): error C2061: syntax error: identifier 'UnlockMutex'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1218): error C2059: syntax error: ';'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1221): error C2059: syntax error: '}'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11t.h(1229): error C2143: syntax error: missing '{' before '*'
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11f.h(24): error C2081: 'C_Initialize': name in formal parameter list illegal
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11f.h(29): error C2091: function returns function
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11f.h(38): error C2081: 'C_Finalize': name in formal parameter list illegal
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11f.h(40): error C2091: function returns function
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11f.h(47): error C2081: 'C_GetInfo': name in formal parameter list illegal
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11f.h(49): error C2091: function returns function
c:\users\usuario\desktop\desarrollo\python-pkcs11\extern\pkcs11f.h(56): error C2081: 'C_GetFunctionList': name in formal parameter list illegal

Looking about this error I saw that some errors are for MS Visual Studio 14.0 (python 3.6 is build with this )
https://docs.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/compiler-error-c2081

Any idea how to solve this ?

[danni]

My guess is the problem is here: master...luisza:winbuild#diff-2eeaed663bd0d25b7e608891384b7298L26

There's a bunch of #defines that are passed into pkcs11[tf].h, which have different representations on different platforms. One of those files should contain the appropriate representations for Visual Studio.

[danni]

Fixed in #33

[jankiraoupada]

I am using python-pkcs11 in one of my python project which is throwing is throwing error in windows however the same works fine in Linux and OSX. Appreciate if anyone can help me here. _pkcs11.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
_loader.c
pkcs11_loader.c(788): fatal error C1083: Cannot open include file: 'dlfcn.h': No such file or directory`

[danni]

@jankiraoupada dlfcn should only be imported on Posix. Are you trying to build in Cygwin? That will lead to whacky type sizes and likely a broken result.

[jankiraoupada]

@jankiraoupada dlfcn should only be imported on Posix. Are you trying to build in Cygwin? That will lead to whacky type sizes and likely a broken result.

[danni]

0.5.0 does not support Windows. You will need to upgrade.

…

On 17 Nov 2020, 10:47 PM +1100, jankiraoupada ***@***.***>, wrote: > > @jankiraoupada dlfcn should only be imported on Posix. Are you trying to build in Cygwin? That will lead to whacky type sizes and likely a broken result. I am trying to install a python project (which uses python-pkcs11 -- 0.5.0 ) in my windows 10, which is developed and tested in OSX and linux. I am simply using python virtual environment to setup execution environment, however getting the above error while installing packages. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

[jankiraoupada]

ws. You will need to upgrade.

Thanks Danny, worked after upgrade.