5 This is various experiments with the panama-foreign abi JEP for
6 invoking native C functions from Java directly and without JNI.
8 The main goal is to experiment with creating a "java friendly" and
9 mostly type-safe api directly in one step, without requiring
12 It uses a gcc plugin to compile c headers to obtain most of the api
13 information, but requires cpp and perl to extract the values of
14 #define constants as they are not available to the gcc plugin.
16 This api information is then converted to Java code using a
17 config-directed perl script `export-api'. The config file can be used
18 to create static (all-in-one) or object oriented output. It includes
19 templated strings as well as perl fragments to generate various source
22 The vulkan api is officially defined by a registry in xml, so there's
23 a separate generator for that.
28 Requirements are gcc and GNU cpp, perl, GNU make, and of course a
29 compatible jdk-foreign jdk. The various modules require headers or
30 sdks for their corresponding libraries, e.g. ffmpeg-5.x,
33 Copy config.make.in to config.make and modify any variables required.
35 Then make everything, parallel make should work.
43 Or run a demo (see next section):
45 $ make run-notzed.vkregistry.test/vulkan.test.TestMandelbrot
47 A non-recursive make setup is used although make file fragments are
48 included from various locations across the modules. All java is
51 JAVA_HOME must point to a compatible panama-enabled jdk.
53 The latest at the time of writing was:
55 branch: foreign-jextract
56 commit: 2e1291680024f12fbf2f0f02b0f79d7b4348b369
57 date: Fri Feb 4 11:01:21 2022 +0000
59 All output and intermediate files are in bin/
61 bin/modules/<module>/classes - compiled java modules
62 bin/gen/<module>/classes - generated java
63 bin/gen/<module>/gen - generated intermediate non-java
65 These are more or less an exploded view of all jmod files:
67 bin/<target>/bin - commands for all modules
68 bin/<target>/lib - libraries/config and modular jar files for all modules
69 bin/<target>/include - header files for all modules
73 bin/<target>/jmods - .jmod modules
79 Most examples have a demo, see the <module>_JMAIN variables in the
80 Makefile for the targets. They are executed using:
82 $ make run-<module>/<main-class>
87 notzed.nativez contains some support classes and the code generator.
88 The gcc plugin source is in src/notzed.nativez/native/ the code
89 generator is in src/notzed.nativez/{bin,lib}.
91 notzed.api is a pseudo-module containing a simple test c api for
92 experiments, it just builds into a library.
94 notzed.apistatic is a 'all in one class' static wrapper for
97 notzed.apiobject is an object-oriented wrapper for notzed.api.
99 notzed.clstatic is an 'all in one class' static wrapper for OpenCL
100 (2.1). The api closesly matches the C api except it converts error
101 codes into exceptins and can infer certain arguments from others such
102 as length parameters. This is probably the most complete api.
104 notzed.ffmpeg is a partial object-oriented mapping for ffmpeg-5.0 that
105 closely follows the jjmpeg design. It's enough to read video frames.
106 The demo requires a file 'movie.api' in the root directory to run.
108 notzed.vkheader uses the header-based generator on the vulkan
109 installed headers. This is still very incomplete work in progress.
110 Much meta-data is lost such as which functions are extensions during
111 the generation of the headers. This is incomplete and dead-ended.
113 notzed.vkregistry uses a completely different generator which directly
114 parses the official xml registry specification for vulkan
115 (/usr/share/vulkan/registry/vk.xml). This is directly converted to
116 about-as-java-friendly a vulkan api as one can hope for, particularly
117 the constructors for all the config objects. This is incomplete and
120 notzed.vulkan also uses a different generator which directly parses
121 the official xml registry specification for vulkan
122 (/usr/share/vulkan/registry/vk.xml). This version uses templates to
123 generate the various structures in more concise and relatively
124 consistenet way. Work in progress.
129 The main generator is written in perl and lives in
130 src/notzed.nativez/{bin,lib}.
134 * run the gcc plugin to extract all of the available c structures from
135 gcc and save them to a perl hash.
137 gcc plugins aren't very well documented so it was a lot of trial and
138 error to get it to output all the type information even starting
139 with a partial example. The names of parameters for function calls
140 were particularly problematic.
142 * optionally run export-defines to extract #define constants. They
143 can be grouped/excluded by name or by the filename they belong to.
144 The the first rule to match will complete the processing for a given
147 This is also a surprisingly difficult process because the c
148 pre-processor can just generate arbitrary c expressions so the only
149 way to find their correct value is to execute the c. So the export
150 script generates a c file which is compiled and executed to generate
151 the perl definitions.
153 Currently the types are mapped to a 'compatible' native type by
154 using the gcc operators __builtin_choose_expr,
155 __builtin_types_compatible_p, and typeof to implement a
156 pseudo-function overloading in c - possiblly using c++ is a better
157 choice here. For now inclusions or exclusions are required to
158 avoid problematic definitions that confuse these operators.
160 These files are then fed to generate-api. It proceeds in multiple
163 First stages are handled by lib/api.pm.
165 * Load and preprocess the api definition.
167 - Perform a bunch of 'fix up' processing on the data structures such
168 as inserting anonymous types which are not generated by the plugin.
170 - Create a full dependency tree for the objects
171 specificallyreferenced by the api definition so only objects and
172 functions of interest are included.
174 - Resolve all the various export options using some rules and store
175 the information about the selected options onto target objects.
177 Then generate-api produces the java files from a combination of the
178 processed data definitions and the api definition.
180 * Export 'libraries'. These are static classes of functions and/or
183 - Can include static methods. Methods can have per-method template
186 - Can include constants.
188 * Export 'structures'. These represent C pointers.
190 - struct/union have a MemorySegment, a layout, and potentially
191 accessors for fields.
193 - anonymous structures just have a MemoryAddress and no accessors.
195 - Can include static and member methods. Methods can have per-method
198 - Can include constants.
200 * Export 'calls'. These are function interfaces for function pointers.
202 - The interface is the java form of the call.
204 - If available a typedef name is used, otherwise the names are mapped
205 to a mangled name of the form Call_<args>_<return>.
207 - An upcall factory creates an upcall to an instance of a hidden
208 trampline interface which performs the mapping of arguments to java
211 - A downcall factory creates a downcall which maps the java call to
214 - Both factories return a record of type FunctionPointer<type> which
215 contains both a NativeSymbol and the java interface instance so they
216 can be used from either environment.
218 * Export 'constants'. These are referenced enums or any defines which
219 haven't been included in any other library or struct.
221 - enums are mapped to an interface with fields of:
222 'static final int FOO = xx'
224 'static final long FOO = xx'
226 - defines are mapped to a matching native type which includes
227 floating point types and strings in addition to all the integral
230 lib/code.pm is used to generate some of the code and handle
231 templating. Common code templates are defined in lib/code.api but can
232 be extended or replaced by a given api file.
234 lib/method.pm is used to generate per-field (struct) and per-argument
235 (function) sub-templates for mapping c to and from java. The
236 sub-templates it uses are defined in lib/types.api but again they can
237 be extended or replaced by a given api file.
242 It's all very much work in progress and due to the constant changes in
243 panama will be in flux for some time.
245 * bitfields are implemented.
246 * varargs is not implemented.
247 * the generator for notzed.vkregistry uses a lot of miserable
249 * the scope and object lifecycle stuff is not really sorted out yet.
250 * the config format and features are still being fiddled with.
251 * the config file isn't really documented.
252 * the build system is still being fiddled with, some of the output
253 directories are mixed up.
254 * linux-amd64 only at this point in time.
259 GNU General Public License, version 3 or later, but see individual
260 file headers for specifics.
265 * https://www.zedzone/software/panamaz.html - project page.
266 * https://openjdk.java.net/projects/panama - openjdk panama page.
267 * https://github.com/openjdk/panama-foreign - openjdk panama source.