Add dependency generation for .api files

[panamaz] / README

Introduction
------------

This is various experiments with the panama-foreign abi JEP for
invoking native C functions from Java directly and without JNI.

The main goal is to experiment with creating a "java friendly" and
mostly type-safe api directly in one step, without requiring
additional wrapping.

It uses a gcc plugin to compile c headers to obtain most of the api
information, but requires cpp and perl to extract the values of
#define constants as they are not available to the gcc plugin.

This api information is then converted to Java code using a
config-directed perl script `export-api'.  The config file can be used
to create static (all-in-one) or object oriented output.  It includes
templated strings as well as perl fragments to generate various source
parts.

The vulkan api is officially defined by a registry in xml, so there's
a separate generator for that.

Compile
-------

Requirements are gcc and GNU cpp, perl, GNU make, and of course a
compatible jdk-foreign jdk.  The various modules require headers or
sdks for their corresponding libraries, e.g. ffmpeg-5.x,
vulkan-tools-1.2.x.

Copy config.make.in to config.make and modify any variables required.

First compile the native binding tools.  This should only need to be
done once after which it auto-updates properly.

$ make notzed.nativez

Then make everything else, parallel make should work.

$ make -j

Or build one module:

$ make notzed.nativez

Or run a demo (see next section):

$ make run-notzed.vkregistry/vulkan.test.TestVulkan

A non-recursive make setup is used although make file fragments are
included from various locations across the modules.  All java is
compiled as modules.

JAVA_HOME must point to a compatible panama-enabled jdk.

The latest at the time of writing was:

 branch: foreign-jextract
 commit: 2e1291680024f12fbf2f0f02b0f79d7b4348b369
 date: Fri Feb 4 11:01:21 2022 +0000

All output and intermediate files are in bin/

bin/modules/<module>/classes - compiled java modules
bin/gen/<module>/classes - generated java
bin/gen/<module>/gen - generated intermediate non-java

These are more or less an exploded view of all jmod files:

bin/<target>/bin - commands for all modules
bin/<target>/lib - libraries/config and modular jar files for all modules
bin/<target>/include - header files for all modules

Finally:

bin/<target>/jmods - .jmod modules


Demos
-----

Most examples have a demo, see the <module>_JMAIN variables in the
Makefile for the targets.  They are executed using:

$ make run-<module>/<main-class>

Modules
--------

notzed.nativez contains some support classes and the code generator.
The gcc plugin source is in src/notzed.nativez/native/ the code
generator is in src/notzed.nativez/{bin,lib}.

notzed.api is a pseudo-module containing a simple test c api for
experiments, it just builds into a library.

notzed.apistatic is a 'all in one class' static wrapper for
notzed.api.

notzed.apiobject is an object-oriented wrapper for notzed.api.

notzed.clstatic is an 'all in one class' static wrapper for OpenCL
(2.1).  The api closesly matches the C api except it converts error
codes into exceptins and can infer certain arguments from others such
as length parameters.  This is probably the most complete api.

notzed.ffmpeg is a partial object-oriented mapping for ffmpeg-5.0 that
closely follows the jjmpeg design.  It's enough to read video frames.
The demo requires a file 'movie.api' in the root directory to run.

notzed.vkheader uses the header-based generator on the vulkan
installed headers.  This is still very incomplete work in progress.
Much meta-data is lost such as which functions are extensions during
the generation of the headers.

notzed.vkregistry uses a completely different generator which directly
parses the official xml registry specification for vulkan
(/usr/share/vulkan/registry/vk.xml).  This is directly converted to
about-as-java-friendly a vulkan api as one can hope for, particularly
the constructors for all the config objects.

Export process
--------------

The main generator is written in perl and lives in
src/notzed.nativez/{bin,lib}.

The process:

* run the gcc plugin to extract all of the available c structures from
  gcc and save them to a perl hash.

  gcc plugins aren't very well documented so it was a lot of trial and
  error to get it to output all the type information even starting
  with a partial example.  The names of parameters for function calls
  were particularly problematic.

* optionally run export-defines to extract #define constants.  They
  can be grouped/excluded by name or by the filename they belong to.
  The the first rule to match will complete the processing for a given
  type.

  This is also a surprisingly difficult process because the c
  pre-processor can just generate arbitrary c expressions so the only
  way to find their correct value is to execute the c.  So the export
  script generates a c file which is compiled and executed to generate
  the perl definitions.

  Currently the types are mapped to a 'compatible' native type by
  using the gcc operators __builtin_choose_expr,
  __builtin_types_compatible_p, and typeof to implement a
  pseudo-function overloading in c - possiblly using c++ is a better
  choice here.  For now inclusions or exclusions are required to
  avoid problematic definitions that confuse these operators.

These files are then fed to generate-api.  It proceeds in multiple
stages.

First stages are handled by lib/api.pm.

* Load and preprocess the api definition.

 - Perform a bunch of 'fix up' processing on the data structures such
   as inserting anonymous types which are not generated by the plugin.

 - Create a full dependency tree for the objects
   specificallyreferenced by the api definition so only objects and
   functions of interest are included.

 - Resolve all the various export options using some rules and store
   the information about the selected options onto target objects.

Then generate-api produces the java files from a combination of the
processed data definitions and the api definition.

* Export 'libraries'.  These are static classes of functions and/or
  constants.

 - Can include static methods.  Methods can have per-method template
   overrides.

 - Can include constants.

* Export 'structures'.  These represent C pointers.

 - struct/union have a MemorySegment, a layout, and potentially
   accessors for fields.

 - anonymous structures just have a MemoryAddress and no accessors.

 - Can include static and member methods.  Methods can have per-method
   template overrides.

 - Can include constants.

* Export 'calls'.  These are function interfaces for function pointers.

 - The interface is the java form of the call.

 - If available a typedef name is used, otherwise the names are mapped
   to a mangled name of the form Call_<args>_<return>.

 - An upcall factory creates an upcall to an instance of a hidden
   trampline interface which performs the mapping of arguments to java
   and returns to c.

 - A downcall factory creates a downcall which maps the java call to
   native and back.

 - Both factories return a record of type FunctionPointer<type> which
   contains both a NativeSymbol and the java interface instance so they
   can be used from either environment.

* Export 'constants'.  These are referenced enums or any defines which
  haven't been included in any other library or struct.

 - enums are mapped to an interface with fields of:
     'static final int FOO = xx'
   of
     'static final long FOO = xx'

 - defines are mapped to a matching native type which includes
   floating point types and strings in addition to all the integral
   types.

lib/code.pm is used to generate some of the code and handle
templating. Common code templates are defined in lib/code.api but can
be extended or replaced by a given api file.

lib/method.pm is used to generate per-field (struct) and per-argument
(function) sub-templates for mapping c to and from java.  The
sub-templates it uses are defined in lib/types.api but again they can
be extended or replaced by a given api file.

Status
------

It's all very much work in progress and due to the constant changes in
panama will be in flux for some time.

* bitfields are implemented.
* varargs is not implemented.
* the generator for notzed.vkregistry uses a lot of miserable
  write-once perl.
* the scope and object lifecycle stuff is not really sorted out yet.
* the config format and features are still being fiddled with.
* the config file isn't really documented.
* the build system is still being fiddled with, some of the output
  directories are mixed up.
* linux-amd64 only at this point in time.

License
-------

GNU General Public License, version 3 or later, but see individual
file headers for specifics.

Links
-----

 * https://www.zedzone/software/panamaz.html - project page.
 * https://openjdk.java.net/projects/panama - openjdk panama page.
 * https://github.com/openjdk/panama-foreign - openjdk panama source.