$250 - Common HL/HXCPP native C API binding format

[larsiusprime]

Introductory Information:

Hashlink is basically the successor Neko in terms of Haxe VM targets, with the added advantage of having both a compiled-C and a bytecode target.

HXCPP is the traditional workhorse of Haxe native C++ targets.

Hashlink promises some advantages to workflow -- particularly faster compilation speed during development, but it's still relatively new and most of the C/C++ bindings for existing libraries for Haxe native targets are still written for HXCPP, and HXCPP is a solid well-maintained target that will surely see continued use alongside HashLink. This puts native extension library maintainers in a dilemma, do we support HXCPP or Hashlink, or both? Right now supporting both means maintaining two parallel sets of bindings, which is not only double the work, but also fertile ground for subtle duplication/oops-I-forgot-to-update-the-other-one errors.

This bounty is for developing a common format for writing native extensions to C API's for Haxe, that is compatible with both HXCPP and HashLink.

This is a separate issue from automatically generating native extensions with minimal effort from, say, raw C files. It's okay for the sake of this bounty if the accepted solution requires a bit of manual labor in writing the bindings themselves, so long as each binding doesn't have to be written twice, once for HXCPP and once for HashLink.

The Project

• Produce a method that bridges a C API to Haxe
• The solution should produce both HL bindings and HXCPP bindings
• The programmer should only have to write an "exposure" for a given C API function once, and get both bindings out of it
• Provide a working proof of concept example of limited scope
• Open source the whole shebang under MIT license or equivalent
• The exposed Haxe interface should match the C interface as closely as possible
• Absolute minimum dependencies
• Account for and note any "leaky abstractions" / incompatibilities between HXCPP & HL
• Minimize use of operations that generate expensive side-effects (such as hidden Dynamics on HXCPP target) -- libraries that rely on this solution should be suitable for high-performance things like physics and rendering.
• Exact approach (e.g. using CFFI, CFFI Prime, C externs, untyped C blocks, etc) is entirely up to you, just provide a reasoning for why your solution is the best one you could think of compared to the alternatives.

The sample should:

• Demonstrate passing two numbers from Haxe, adding them together on the C side, and returning them to Haxe
• Demonstrate passing a file name from Haxe, loading the file and resolving it as raw bytes on the C side, and returning the bytes to Haxe
• Demonstrate passing two (ASCII) strings from Haxe, concatenate them together on the C side, and return them to Haxe
• Demonstrate all the above, with the functions implemented in loose .c and/or .h files
• Demonstrate all the above, with the functions implemented in a .dll file
• Be able to be compiled and run against hxcpp target and hl target with a simple .hxml script

Out of scope

None of these things are part of the project:

• C++ API support (we only care about supporting plain C API's for now, save this for later)
• Exhaustive documentation (just a simple example and a quick description is fine)
• Unicode string support in the sample (just ASCII is fine for now)
• Pass a structured object from Haxe, mutate some of its values on the C side, and return it to Haxe
• Properly handle a C api that asks for a C data structure as input (important to solve eventually but let's not worry about it just yet)
• Automatic generation of bindings from a minimal description (such as c header files alone or whatever)

Prerequisites

• Knowledge of HXCPP
• Knowledge of Hashlink
• Knowledge of Macros
• Knowledge of C

What's been done so far

Nothing specifically on this project, but here's some prior work to learn form:

• Linc -- solely focused on HXCPP
• Everything I know about CFFI / CFFI Prime
• Haxe externs
• Hashlink links:
  • https://github.com/HaxeFoundation/hashlink/wiki/C-API-Documentation
  • https://haxe.org/blog/hashlink-indepth/
• Updates from comments:
  • Nadako's JNI stuff
  • Nicolas' WebIDL (generate emscriptem/wasm+HL from IDL files)

How Much can Lars help?

• I'm available to talk to and ask/answer questions but you'll be doing the majority if not all the work here. I recognize I'm out of my depth.

What remains to be done

• The whole thing!

Budget/Timeline:

• No specific timeline. 1 month would be great? I'd like to see it done before November.

Budget is $100 USD. $200 USD $250 USD

[nadako]

For the people looking into this:

I experimented with auto-generating glue for Java's JNI API for hashlink before and had some success. The idea is pretty simple: we define an extern class in Haxe like this and then process it with a macro and generate glue C code somewhat like this. That particular code is purely experimental and JNI-specific of course, but I think the idea is really nice and the same approach can be used for generating both HL and HXCPP glue code.

[ncannasse]

I have also some experiment to generate both Emscriptem and HL bindings from an IDL structure, see https://github.com/ncannasse/webidl

[larsiusprime]

@ncannasse -- very interesting, Nicolas! Do you actually use it in production anywhere or is it just tinkering for now? Presumably this format could be used to generate hxcpp bindings in addition to your emscriptem/HL bindings?

[ncannasse]

@larsiusprime not yet in production, but I did use it for some Bullet3D experiments here https://github.com/HaxeFoundation/hashlink/tree/master/libs/bullet

The nice thing it that it generates both Haxe externs (using macros) and C stub code from IDL.
Cons is that it only supports C++ for now, but I guess the IDL could be enriched with additional annotations to support C.

One big cons versus "manual" wrapper is that it does not allow much cooperation with GC so it requires for the Haxe wrapper objects to keep references on the C++ objects. And when the the Haxe object gets GC'ed to delete the C++ object automatically. This also requires to make sure that Haxe references are keeping each other the way C++ objects are, which is extra post wrapping work.

[larsiusprime]

@ncannasse speaking of "manual" wrapper -- is it possible in principle to write the "manual" part, with GC fiddling and all, and have a script/macro/etc generate both a HXCPP and a Hashlink binding from whatever intermediate format is used? Or is the way GC behaves in Hashlink and HXCPP sufficiently different to make this not possible?

[kevinresol]

Any ways to pool a budget? Maybe something like OpenBounty?

[ncannasse]

I think the problem of "automatic" wrapper generation VS an IDL and the problem of a "common" way to express the wrappers are two separate problems. Once the "common" problem is solved, it will help a lot the "automatic" problem.

Regarding a good way to express the "common" wrapper generation, there are two ways to deal with it:

a) write C code with an #include <common_wrapper.h> that defines an API that behind the scene will call HL API or HxCPP one (or other platforms). But this is quite low level and I'm not sure how much additional platform-specific things will be required there.

b) write Haxe code that also implements a cross platform "extern" API, that does not get compiled but that gets read by macros to output the corresponding platform-specific C code. That's slightly higher level and allows to add platform specific constructors automatically by detecting the places they are needed from the Haxe AST.

[larsiusprime]

@kevinresol -- possibly! So far I've just done everything informally. Rest assured, if several people contribute something significant I'll do my best to split the bounty fairly, and maybe even bump it up a little if that feels necessary.

[kevinresol]

I mean the other way round, I probably would like to contribute some money to the budget

[larsiusprime]

@kevinresol -- oh, interesting! That's a swell idea, can you link to any services you like? What's this OpenBounty thing for instance?

[kevinresol]

Not really, I just randomly searched it without deeply looking at it yet. Maybe we should discuss in a new thread.

[larsiusprime]

@kevinresol:
Great, let's discuss that here.

@ncannasse:
Thanks for the input! Re this:

I think the problem of "automatic" wrapper generation VS an IDL and the problem of a "common" way to express the wrappers are two separate problems. Once the "common" problem is solved, it will help a lot the "automatic" problem.

I'll update the issue to indicate the scope of this bounty is solely the "common" problem. If we can solve that, we can tackle the "automatic" problem as a separate issue.

[kevinresol]

How about extracting (one of) the libs into a separate repo to demonstrate how to write a native lib for hl?

Take mysql as an example, move all the mysql source, glue code, haxe extern files, build instructions (cmake) into a repo which can be consumed by simply adding sth like -lib hl-mysql to a hl project. This is similar to what linc does. I actually like linc because I found its self-contained approach makes it much easier to develop and consume native extensions.

[kevinresol]

How about extracting (one of) the libs into a separate repo to demonstrate how to write a native lib for hl?

Take mysql as an example, move all the mysql source, glue code, haxe extern files, build instructions (cmake) into a repo which can be consumed by simply adding sth like -lib hl-mysql to a hl project. This is similar to what linc does. I actually like linc because I found its self-contained approach makes it much easier to develop and consume native extensions.

@ncannasse what's your opinion on this?