Introduction ------------ This is various experiments with the panama-foreign abi JEP for invoking native C functions from Java directly and without JNI. The main goal is to experiment with creating a "java friendly" and mostly type-safe api directly in one step, without requiring additional wrapping. It uses a gcc plugin to compile c headers to obtain most of the api information, but requires cpp and perl to extract the values of #define constants as they are not available to the gcc plugin. This api information is then converted to Java code using a config-directed perl script `export-api'. The config file can be used to create static (all-in-one) or object oriented output. It includes templated strings as well as perl fragments to generate various source parts. The vulkan api is officially defined by a registry in xml, so there's a separate generator for that. Compile ------- Requirements are gcc and GNU cpp, perl, GNU make, and of course a compatible jdk-foreign jdk. The various modules require headers or sdks for their corresponding libraries, e.g. ffmpeg-5.x, vulkan-tools-1.2.x. Copy config.make.in to config.make and modify any variables required. Then make everything, parallel make should work. $ make -j Or build one module: $ make notzed.nativez Or run a demo (see next section): $ make run-notzed.vkregistry.test/vulkan.test.TestMandelbrot A non-recursive make setup is used although make file fragments are included from various locations across the modules. All java is compiled as modules. JAVA_HOME must point to a compatible panama-enabled jdk. The latest at the time of writing was: branch: foreign-jextract commit: 2e1291680024f12fbf2f0f02b0f79d7b4348b369 date: Fri Feb 4 11:01:21 2022 +0000 All output and intermediate files are in bin/ bin/modules//classes - compiled java modules bin/gen//classes - generated java bin/gen//gen - generated intermediate non-java These are more or less an exploded view of all jmod files: bin//bin - commands for all modules bin//lib - libraries/config and modular jar files for all modules bin//include - header files for all modules Finally: bin//jmods - .jmod modules Demos ----- Most examples have a demo, see the _JMAIN variables in the Makefile for the targets. They are executed using: $ make run-/ Modules -------- notzed.nativez contains some support classes and the code generator. The gcc plugin source is in src/notzed.nativez/native/ the code generator is in src/notzed.nativez/{bin,lib}. notzed.api is a pseudo-module containing a simple test c api for experiments, it just builds into a library. notzed.apistatic is a 'all in one class' static wrapper for notzed.api. notzed.apiobject is an object-oriented wrapper for notzed.api. notzed.clstatic is an 'all in one class' static wrapper for OpenCL (2.1). The api closesly matches the C api except it converts error codes into exceptins and can infer certain arguments from others such as length parameters. This is probably the most complete api. notzed.ffmpeg is a partial object-oriented mapping for ffmpeg-5.0 that closely follows the jjmpeg design. It's enough to read video frames. The demo requires a file 'movie.api' in the root directory to run. notzed.vkheader uses the header-based generator on the vulkan installed headers. This is still very incomplete work in progress. Much meta-data is lost such as which functions are extensions during the generation of the headers. This is incomplete and dead-ended. notzed.vkregistry uses a completely different generator which directly parses the official xml registry specification for vulkan (/usr/share/vulkan/registry/vk.xml). This is directly converted to about-as-java-friendly a vulkan api as one can hope for, particularly the constructors for all the config objects. This is incomplete and dead-ended. notzed.vulkan also uses a different generator which directly parses the official xml registry specification for vulkan (/usr/share/vulkan/registry/vk.xml). This version uses templates to generate the various structures in more concise and relatively consistenet way. Work in progress. Export process -------------- The main generator is written in perl and lives in src/notzed.nativez/{bin,lib}. The process: * run the gcc plugin to extract all of the available c structures from gcc and save them to a perl hash. gcc plugins aren't very well documented so it was a lot of trial and error to get it to output all the type information even starting with a partial example. The names of parameters for function calls were particularly problematic. * optionally run export-defines to extract #define constants. They can be grouped/excluded by name or by the filename they belong to. The the first rule to match will complete the processing for a given type. This is also a surprisingly difficult process because the c pre-processor can just generate arbitrary c expressions so the only way to find their correct value is to execute the c. So the export script generates a c file which is compiled and executed to generate the perl definitions. Currently the types are mapped to a 'compatible' native type by using the gcc operators __builtin_choose_expr, __builtin_types_compatible_p, and typeof to implement a pseudo-function overloading in c - possiblly using c++ is a better choice here. For now inclusions or exclusions are required to avoid problematic definitions that confuse these operators. These files are then fed to generate-api. It proceeds in multiple stages. First stages are handled by lib/api.pm. * Load and preprocess the api definition. - Perform a bunch of 'fix up' processing on the data structures such as inserting anonymous types which are not generated by the plugin. - Create a full dependency tree for the objects specificallyreferenced by the api definition so only objects and functions of interest are included. - Resolve all the various export options using some rules and store the information about the selected options onto target objects. Then generate-api produces the java files from a combination of the processed data definitions and the api definition. * Export 'libraries'. These are static classes of functions and/or constants. - Can include static methods. Methods can have per-method template overrides. - Can include constants. * Export 'structures'. These represent C pointers. - struct/union have a MemorySegment, a layout, and potentially accessors for fields. - anonymous structures just have a MemoryAddress and no accessors. - Can include static and member methods. Methods can have per-method template overrides. - Can include constants. * Export 'calls'. These are function interfaces for function pointers. - The interface is the java form of the call. - If available a typedef name is used, otherwise the names are mapped to a mangled name of the form Call__. - An upcall factory creates an upcall to an instance of a hidden trampline interface which performs the mapping of arguments to java and returns to c. - A downcall factory creates a downcall which maps the java call to native and back. - Both factories return a record of type FunctionPointer which contains both a NativeSymbol and the java interface instance so they can be used from either environment. * Export 'constants'. These are referenced enums or any defines which haven't been included in any other library or struct. - enums are mapped to an interface with fields of: 'static final int FOO = xx' of 'static final long FOO = xx' - defines are mapped to a matching native type which includes floating point types and strings in addition to all the integral types. lib/code.pm is used to generate some of the code and handle templating. Common code templates are defined in lib/code.api but can be extended or replaced by a given api file. lib/method.pm is used to generate per-field (struct) and per-argument (function) sub-templates for mapping c to and from java. The sub-templates it uses are defined in lib/types.api but again they can be extended or replaced by a given api file. Status ------ It's all very much work in progress and due to the constant changes in panama will be in flux for some time. * bitfields are implemented. * varargs is not implemented. * the generator for notzed.vkregistry uses a lot of miserable write-once perl. * the scope and object lifecycle stuff is not really sorted out yet. * the config format and features are still being fiddled with. * the config file isn't really documented. * the build system is still being fiddled with, some of the output directories are mixed up. * linux-amd64 only at this point in time. License ------- GNU General Public License, version 3 or later, but see individual file headers for specifics. Links ----- * https://www.zedzone/software/panamaz.html - project page. * https://openjdk.java.net/projects/panama - openjdk panama page. * https://github.com/openjdk/panama-foreign - openjdk panama source.