Remove some internal dumps.

[panamaz] / README

diff --git a/README b/README
index 6fe7c06..3112661 100644 (file)
--- a/README
+++ b/README
@@ -2,91 +2,256 @@
 Introduction
 ------------
 
 Introduction
 ------------
 

-This is an alternative to jextract for the OpenJDK panama project
-which in turn is a project to implement Java to Native interfaces at
-runtime.
+This is various experiments with the panama-foreign abi JEP for
+invoking native C functions from Java directly and without JNI.

 
 

-It uses a gcc plugin for type extraction and a "simple" perl script to
-transform this into Java source files with the appropriate annotations
-for panama to do it's thing.
+The main goal is to experiment with creating a "java friendly" and
+mostly type-safe api directly in one step, without requiring
+additional wrapping.
+
+It uses a gcc plugin to compile c headers to obtain most of the api
+information, but requires cpp and perl to extract the values of
+#define constants as they are not available to the gcc plugin.
+
+This api information is then converted to Java code using a
+config-directed perl script `export-api'.  The config file can be used
+to create static (all-in-one) or object oriented output.  It includes
+templated strings as well as perl fragments to generate various source
+parts.
+
+The vulkan api is officially defined by a registry in xml, so there's
+a separate generator for that.

 
 Compile
 -------
 
 
 Compile
 -------
 

-Edit the makefile to point to your gcc directory and make.
+Requirements are gcc and GNU cpp, perl, GNU make, and of course a
+compatible jdk-foreign jdk.  The various modules require headers or
+sdks for their corresponding libraries, e.g. ffmpeg-5.x,
+vulkan-tools-1.2.x.

 
 

-$ cd src
-$ vi Makefile
-$ make
+Copy config.make.in to config.make and modify any variables required.

 
 

-generate is a perl script.
+Then make everything, parallel make should work.

 
 

-See test-api/* for a simple example.  JAVA_HOME must point to
-a compatible panama-enabled jdk.
+$ make -j

 
 

-Use dump.so
------------
+Or build one module:

 
 

-dump.so is a gcc plugin which captures structures, unions, enums,
-exported functions, and a few typedefs into a perl source file
-containing a single hash.
+$ make notzed.nativez

 
 

-Put all the include files required for your api in api.h.
+Or run a demo (see next section):

 
 

-$ gcc -fplugin=src/export.so -fplugin-arg-export-output=api.pm api.h -o /dev/null
+$ make run-notzed.vkregistry.test/vulkan.test.TestMandelbrot

 
 

-api.pm is a text file of perl source-code which contains a descriptor
-of all supported types found.
+A non-recursive make setup is used although make file fragments are
+included from various locations across the modules.  All java is
+compiled as modules.

 
 

-Use generate
-------------
+JAVA_HOME must point to a compatible panama-enabled jdk.

 
 

-Quick and dirty:
+The latest at the time of writing was:

 
 

-$ generate -d dir -t name.space -c APILib -lapi ./api.pm
+ branch: foreign-jextract
+ commit: 2e1291680024f12fbf2f0f02b0f79d7b4348b369
+ date: Fri Feb 4 11:01:21 2022 +0000

 
 

-Generate all structures in api.pm (the ./ is required) and place all
-functions into dir/name/space/APILib.java which will link with
-libapi.so.
+All output and intermediate files are in bin/

 
 

-The generator is quite flexible, sub-sets of functions can be placed
-in alternative *Lib.java files, types can be filtered.  If a function
-or type is specified explicitly, then only those types it requires are
-included in the generated output (recursively).
+bin/modules/<module>/classes - compiled java modules
+bin/gen/<module>/classes - generated java
+bin/gen/<module>/gen - generated intermediate non-java

 
 

-See the top of the file for details, it's probably up to date.
+These are more or less an exploded view of all jmod files:

 
 

-Status
-------
+bin/<target>/bin - commands for all modules
+bin/<target>/lib - libraries/config and modular jar files for all modules
+bin/<target>/include - header files for all modules
+
+Finally:
+
+bin/<target>/jmods - .jmod modules
+
+
+Demos
+-----
+
+Most examples have a demo, see the <module>_JMAIN variables in the
+Makefile for the targets.  They are executed using:
+
+$ make run-<module>/<main-class>
+
+Modules
+--------
+
+notzed.nativez contains some support classes and the code generator.
+The gcc plugin source is in src/notzed.nativez/native/ the code
+generator is in src/notzed.nativez/{bin,lib}.
+
+notzed.api is a pseudo-module containing a simple test c api for
+experiments, it just builds into a library.
+
+notzed.apistatic is a 'all in one class' static wrapper for
+notzed.api.
+
+notzed.apiobject is an object-oriented wrapper for notzed.api.
+
+notzed.clstatic is an 'all in one class' static wrapper for OpenCL
+(2.1).  The api closesly matches the C api except it converts error
+codes into exceptins and can infer certain arguments from others such
+as length parameters.  This is probably the most complete api.
+
+notzed.ffmpeg is a partial object-oriented mapping for ffmpeg-5.0 that
+closely follows the jjmpeg design.  It's enough to read video frames.
+The demo requires a file 'movie.api' in the root directory to run.
+
+notzed.vkheader uses the header-based generator on the vulkan
+installed headers.  This is still very incomplete work in progress.
+Much meta-data is lost such as which functions are extensions during
+the generation of the headers.  This is incomplete and dead-ended.
+
+notzed.vkregistry uses a completely different generator which directly
+parses the official xml registry specification for vulkan
+(/usr/share/vulkan/registry/vk.xml).  This is directly converted to
+about-as-java-friendly a vulkan api as one can hope for, particularly
+the constructors for all the config objects.  This is incomplete and
+dead-ended.
+
+notzed.vulkan also uses a different generator which directly parses
+the official xml registry specification for vulkan
+(/usr/share/vulkan/registry/vk.xml).  This version uses templates to
+generate the various structures in more concise and relatively
+consistenet way.  Work in progress.
+
+Export process
+--------------
+
+The main generator is written in perl and lives in
+src/notzed.nativez/{bin,lib}.
+
+The process:
+
+* run the gcc plugin to extract all of the available c structures from
+  gcc and save them to a perl hash.

 
 

-This is currently just a 'quick and dirty' weekend hack.  The tools
-spew a lot of debug.  dump.cc is basic c-compatible-c++, because I
-don't know c++.
+  gcc plugins aren't very well documented so it was a lot of trial and
+  error to get it to output all the type information even starting
+  with a partial example.  The names of parameters for function calls
+  were particularly problematic.

 
 

-dump.so works by hooking into the gcc pre-compiled headers function,
-it might work with a c source file as well.  It's only been used with
-gcc 9.
+* optionally run export-defines to extract #define constants.  They
+  can be grouped/excluded by name or by the filename they belong to.
+  The the first rule to match will complete the processing for a given
+  type.

 
 

-The dumper will hard-fail if it sees defintions it doesn't understand,
-and currently it's only been tested with a small nubmer of libraries.
+  This is also a surprisingly difficult process because the c
+  pre-processor can just generate arbitrary c expressions so the only
+  way to find their correct value is to execute the c.  So the export
+  script generates a c file which is compiled and executed to generate
+  the perl definitions.

 
 

-Enumerations are currently translated to integers.  The generator
-doesn't currently export any of the symbolic values to Java but it is
-easy to add.
+  Currently the types are mapped to a 'compatible' native type by
+  using the gcc operators __builtin_choose_expr,
+  __builtin_types_compatible_p, and typeof to implement a
+  pseudo-function overloading in c - possiblly using c++ is a better
+  choice here.  For now inclusions or exclusions are required to
+  avoid problematic definitions that confuse these operators.

 
 

-varags is not implemented.
+These files are then fed to generate-api.  It proceeds in multiple
+stages.

 
 

-All function pointers are translated to a unique set of functional
-interfaces whose names are based on the function signature.  So the
-type names are ugly but they are stable.
+First stages are handled by lib/api.pm.

 
 

-Anonymous type names are based on the structure member they belong to
-so they may not be unique.
+* Load and preprocess the api definition.
+
+ - Perform a bunch of 'fix up' processing on the data structures such
+   as inserting anonymous types which are not generated by the plugin.
+
+ - Create a full dependency tree for the objects
+   specificallyreferenced by the api definition so only objects and
+   functions of interest are included.
+
+ - Resolve all the various export options using some rules and store
+   the information about the selected options onto target objects.
+
+Then generate-api produces the java files from a combination of the
+processed data definitions and the api definition.
+
+* Export 'libraries'.  These are static classes of functions and/or
+  constants.
+
+ - Can include static methods.  Methods can have per-method template
+   overrides.
+
+ - Can include constants.
+
+* Export 'structures'.  These represent C pointers.
+
+ - struct/union have a MemorySegment, a layout, and potentially
+   accessors for fields.
+
+ - anonymous structures just have a MemoryAddress and no accessors.
+
+ - Can include static and member methods.  Methods can have per-method
+   template overrides.
+
+ - Can include constants.
+
+* Export 'calls'.  These are function interfaces for function pointers.
+
+ - The interface is the java form of the call.
+
+ - If available a typedef name is used, otherwise the names are mapped
+   to a mangled name of the form Call_<args>_<return>.
+
+ - An upcall factory creates an upcall to an instance of a hidden
+   trampline interface which performs the mapping of arguments to java
+   and returns to c.
+
+ - A downcall factory creates a downcall which maps the java call to
+   native and back.
+
+ - Both factories return a record of type FunctionPointer<type> which
+   contains both a NativeSymbol and the java interface instance so they
+   can be used from either environment.
+
+* Export 'constants'.  These are referenced enums or any defines which
+  haven't been included in any other library or struct.
+
+ - enums are mapped to an interface with fields of:
+     'static final int FOO = xx'
+   of
+     'static final long FOO = xx'
+
+ - defines are mapped to a matching native type which includes
+   floating point types and strings in addition to all the integral
+   types.
+
+lib/code.pm is used to generate some of the code and handle
+templating. Common code templates are defined in lib/code.api but can
+be extended or replaced by a given api file.
+
+lib/method.pm is used to generate per-field (struct) and per-argument
+(function) sub-templates for mapping c to and from java.  The
+sub-templates it uses are defined in lib/types.api but again they can
+be extended or replaced by a given api file.
+
+Status
+------

 
 

-It will only ever support c.
+It's all very much work in progress and due to the constant changes in
+panama will be in flux for some time.

 
 

-Only tested against the panama 'openjdk-14-panama+1-15_linux' on
-slackware-current @ 2020.01.01 and gcc 9.2.0.
+* bitfields are implemented.
+* varargs is not implemented.
+* the generator for notzed.vkregistry uses a lot of miserable
+  write-once perl.
+* the scope and object lifecycle stuff is not really sorted out yet.
+* the config format and features are still being fiddled with.
+* the config file isn't really documented.
+* the build system is still being fiddled with, some of the output
+  directories are mixed up.
+* linux-amd64 only at this point in time.

 
 License
 -------
 
 License
 -------


@@ -99,3 +264,4 @@ Links
 
  * https://www.zedzone/software/panamaz.html - project page.
  * https://openjdk.java.net/projects/panama - openjdk panama page.
 
  * https://www.zedzone/software/panamaz.html - project page.
  * https://openjdk.java.net/projects/panama - openjdk panama page.

+ * https://github.com/openjdk/panama-foreign - openjdk panama source.