Blog Home
Next Gen Native Interfaces
The Island Of The JVM
Over many years the Java virtual machine has shown itself over to be an incredibly powerful and stable system for developing and deploying software. In part, this has come at a cost of the ecosystem being a bit of a walled garden where interfacing with libraries written in languages other than Java has been neglected. The cost of developing such interfaces has also historically been high. An effect of this situation is that we have access to fewer great libraries in our ecosystem, and instead we have some translated Java analogues that are poorly written, architecturally awkward, or have important missing pieces. This has been especially painful with advanced, difficult to replicate libraries in the area of machine learning.
Luckily, there is growing interest in the Java community with so-called 'native interfaces' for running code written in other languages. Historically, these sorts of systems are often called Foreign Function Interfaces. In all cases, they tend to involve:
- Allocating memory that lies outside the purview of the garbage collector -- termed 'off-heap' memory by Java developers. Microsoft's calls this 'managed memory', but in our case we call it 'jvm-heap memory' vs. 'native-heap memory'
- Defining a particular layout of this memory used for communicating structs
- Using OS facilities to find host libraries and to find symbols within those libraries
- Facilities for converting a symbol in a library to something that can be called directly as a function
- A facility for exposing a function as a C pointer that can be called from a C-based system
C rules and semantics (the C ABI) defines a set of rules for interfacing between systems that share the same memory space; a sort of lowest common denominator that we can all agree on.
The Java specification originally included no runtime facilities, and only extremely verbose (but also quite stable) compile-time facilities, for interfacing with C systems. This means that you could not create a cross platform java library that would just work even if the consumer of your code had a library of interest installed. You had to have a compiled portion of your jar built just for their operating system and processor architecture in order to run.
Over time the JVM ecosystem evolved to contain at least 2 systems for dynamically binding to libraries. These are the older Java Native Access (JNA) and the newer Java Native Runtime (JNR). Both were based on libffi - a venerated C library for creating C-based closures that we have mentioned before in some of our past blogs.
Recently, with the JDK-16 looming on the horizon there is now a totally separate FFI pathway in the form of JEP-191. Beyond that, the GraalVM project contains two FFI pathways; a compile-time one for Graal Native and a dynamic one called Truffle Native Function Interface or TNFI for both GraalVM and Graal Native pathways. Of course, for the joy of complication, neither JNA nor JNR work with Graal due to their dependence upon load time dynamic class generation.
That means that if you want to build something on a JVM that works with C libraries you have potentially four (or maybe more!) completely different interfaces that you will need to use depending on the underlying platform you wish to run on.
This is not great, but can be addressed, read on!
Moreover, ideally we would have dynamic binding that is robust to differences in the libraries we are using. We don't want to have a single library upgrade necessitate updates all over the place; we instead would like to simply allow our software to adapt with minimal or no changes. What we have built already works this way, and allows a single version of libpython-clj to work across Python versions 3.5-3.9+. And allows that same library to work when embedded within a Python host via javabridge. And also allows that library to work with Python environments such as pyenv and Conda.
Our Dream Foreign Function Interface
So what does an ideal foreign function interface look like? What are we looking for here?
Just a few basic things:
- Dynamic - When developing the bindings we want them to be fluid; we want to be able to add/remove bindings quickly and be able to use the resulting functions immediately without long compile/build/run/test cycle times.
- Cross Platform - We would like to be able to use different FFI implementations in as many contexts as reasonably possible. This means taking one definition of our library and late binding to the underlying platform. When using Graal Native, our dynamic binding system needs to be compiled directly into concrete classes with attributes that the Graal compiler can use to find the necessary headers and actual system libraries. Doing this means that we can generate bindings for 32 bit systems transparently if we detect our underlying runtime is 32 bits.
- Simple - this stuff is hard to get right. The more time we spend on unnecessary details the less time we spend making sure the tricky bits of the system we are building are exactly right and binary interfaces to C libraries is an area where the costs of not being exactly right can be high.
With this in mind we came up with a data-driven pathway that we can use to drive code generation to bind to a particular FFI implementation. As usual, a data oriented approach, is more flexible, gives us visibility, and simplifies the implementation in the end.
We are now going to walk through an example of using our bindings with JNA, JDK-16, and finally when building a Graal Native executable. Remember, the win here is that through our FFI, you gain access to all of these native interfaces, with little or no changes in your host library.
Starting With The Basics
So let's start with some simple bindings to functions that we can find linked into the JVM executable itself (memset and memcpy). As a side benefit, we will additionally see how incredibly simple it is to build a new Clojure library that we can then share with our friends.
chrisn@chrisn-lt-01:~/dev$ mkdir clj-ffi && cd clj-ffi
Copy this code into a file named 'deps.edn':
{:paths ["src"]
:deps {cnuernber/dtype-next {:mvn/version "6.24"}
net.java.dev.jna/jna {:mvn/version "5.8.0"}}}
Now we start editing src/libc.clj in our source directory.
(ns libc
(:require [tech.v3.datatype.ffi :as dt-ffi]))
(def fn-defs
{:memset {:rettype :pointer
:argtypes [['buffer :pointer]
['byte-value :int32]
['n-bytes :size-t]]}})
We have defined a memset function definition that corresponds to the memset function found in the C header file. This is a pure-data definition so we want to create a library definition that binds our data definition to a concrete library implementation with the default being JNA.
From our Clojure REPL:
libc> (def library-def (dt-ffi/define-library fn-defs))
#'libc/library-def
First is our actual definition. At this point it is hard-bound to JNA. We can now create a library instance by combining our library definition with a string library name or 'nil' to use the current executable:
libc> (def library-instance (dt-ffi/instantiate-library library-def nil))
#'libc/library-instance
This library instance contains both member functions which are strongly typed and it contains a map of function name -> clojure.lang.IFn implementation which we can call. We can get to the map by dereferencing the library:
libc> @library-instance
{:memset #object[tech.v3.datatype.ffi.jna.G__15299$invoker_memset 0x28e55102 "tech.v3.datatype.ffi.jna.G__15299$invoker_memset@28e55102"]}
libc> (def memset (:memset @library-instance))
#'libc/memset
In order to test out our function we will need some native memory:
libc> (require '[tech.v3.datatype :as dtype])
nil
libc> (def nbuf (dtype/make-container :native-heap :float32 (range 10)))
Mar 22, 2021 12:49:05 PM clojure.tools.logging$eval6400$fn__6403 invoke
INFO: Reference thread starting
#'libc/nbuf
libc> nbuf
#native-buffer@0x00007F8404754350<float32>[10]
[0.000, 1.000, 2.000, 3.000, 4.000, 5.000, 6.000, 7.000, 8.000, 9.000]
Which we can now memset to zero:
libc> (memset nbuf 0 40)
#object[tech.v3.datatype.ffi.Pointer 0x5f9fac0 "{:address 0x00007F8404754350 }"]
libc> nbuf
#native-buffer@0x00007F8404754350<float32>[10]
[0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000]
Nice.
Adding a Bit More Automation
We want to add another function, memcpy -- but that means redoing the steps above. Since this process repeats quite a lot though the development of our library we want to add in a few pieces that will help in that automation. In our source - libc.clj we create a library-singleton and make sure that every time this file is compiled we reset it.
(defonce lib (dt-ffi/library-singleton #'fn-defs))
(dt-ffi/library-singleton-reset! lib)
This defines the library only once but it will reset the library every time we recompile our file so if we change our function definitions above our singleton will automatically redefine the library and re-create the library instance.
In our REPL we set the library to be nil in order to cause the underlying system to search for the symbols in the current executable:
libc> (dt-ffi/library-singleton-set! lib nil)
#<G__17468@3656d57c:
{:memset #object[tech.v3.datatype.ffi.jna.G__17468$invoker_memset 0x1dfb3a8 "tech.v3.datatype.ffi.jna.G__17468$invoker_memset@1dfb3a8"]}>
We want to define exposed functions like our memcpy def above so from outside this file our clients can just rely on the memcpy symbol and not on dynamically dereferencing our library instance. In our libc.clj file we type:
(dt-ffi/define-library-functions
libc/fn-defs
(fn [fn-name] (dt-ffi/library-singleton-find-fn lib fn-name))
nil)
If we recompile our libc file, we now have memset defined as a symbol in our namespace:
;; (recompile libc.clj)
libc> (ns-publics 'libc)
...
memset #'libc/memset,
...
So let's define a new function, memcpy. Change the definition of fn-defs to:
(def fn-defs
{:memset {:rettype :pointer
:argtypes [['buffer :pointer]
['byte-value :int32]
['n-bytes :size-t]]}
:memcpy {:rettype :pointer
;;dst src size-t
:argtypes [['dst :pointer]
['src :pointer]
['n-bytes :size-t]]}})
If we now recompile our file we find that memcpy is a public namespace symbol:
;;recompile libc namespace
libc> (def nb-buf (dtype/make-container :native-heap :float32 (range 10)))
#'libc/nb-buf
libc> nb-buf
#native-buffer@0x00007F8404D71BB0<float32>[10]
[0.000, 1.000, 2.000, 3.000, 4.000, 5.000, 6.000, 7.000, 8.000, 9.000]
libc> (memcpy nb-buf nbuf 40)
#object[tech.v3.datatype.ffi.Pointer 0x573b2c9a "{:address 0x00007F8404D71BB0 }"]
libc> nb-buf
#native-buffer@0x00007F8404D71BB0<float32>[10]
[0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000]
libc>
This is great! It means we can iteratively develop our native bindings to our library using the same flow we know and love from traditional Clojure development. Recompiling the namespace generates new function bindings based on the data definition. We used dynamically generated JNA Direct Mapping for all the bindings which means they are about as efficient as they can be with JNA.
Trying Out JDK-16
The same system can be exercised with JDK-16's new FFI layer.
First we have to install and activate a version of JDK-16. The dtype-next repository has both scripts so now I expect that you will clone that repo and from the source directory run 'scripts/activate-jdk16':
chrisn@chrisn-lt-01:~/dev/clj-ffi$ git clone https://github.com/cnuernber/dtype-next
Cloning into 'dtype-next'...
remote: Enumerating objects: 1059, done.
remote: Counting objects: 100% (1059/1059), done.
remote: Compressing objects: 100% (526/526), done.
remote: Total 6782 (delta 852), reused 653 (delta 468), pack-reused 5723
Receiving objects: 100% (6782/6782), 1.53 MiB | 1.36 MiB/s, done.
Resolving deltas: 100% (4864/4864), done.
chrisn@chrisn-lt-01:~/dev/clj-ffi$ cd dtype-next
chrisn@chrisn-lt-01:~/dev/clj-ffi/dtype-next$ source scripts/enable-jdk16
...
chrisn@chrisn-lt-01:~/dev/clj-ffi/dtype-next$ java -version
openjdk version "16-ea" 2021-03-16
OpenJDK Runtime Environment (build 16-ea+32-2190)
OpenJDK 64-Bit Server VM (build 16-ea+32-2190, mixed mode, sharing)
Now we have an early release version of JDK-16 running. We will need a deps.edn profile in order to activate the foreign function JEP with some command line parameters. Move back to your main directory and change your deps.edn to include a new alias:
{:paths ["src"]
:deps {cnuernber/dtype-next {:mvn/version "6.24"}
net.java.dev.jna/jna {:mvn/version "5.8.0"}}
:aliases
{:jdk-16
{:jvm-opts ["--add-modules" "jdk.incubator.foreign" "-Dforeign.restricted=permit" "--add-opens" "java.base/java.lang=ALL-UNNAMED" "-Djava.library.path=/usr/lib/x86_64-linux-gnu"]}}}
Let's start a new REPL in the same terminal where you enabled JDK-16:
chrisn@chrisn-lt-01:~/dev/clj-ffi$ clj -A:jdk-16
WARNING: Using incubator modules: jdk.incubator.foreign
Clojure 1.10.2
user=> (require '[libc :as libc])
nil
user=> (require '[tech.v3.datatype :as dtype])
nil
user=> (require '[tech.v3.datatype.ffi :as dt-ffi])
nil
user=> (dt-ffi/set-ffi-impl! :jdk)
...
user=> (dt-ffi/library-singleton-set! libc/lib nil)
#object[tech.v3.datatype.ffi.mmodel.G__9866 0x54d8236c {:status :ready, :val {:memset #object[tech.v3.datatype.ffi.mmodel.G__9866$invoker_memset 0x5697f801 "tech.v3.datatype.ffi.mmodel.G__9866$invoker_memset@5697f801"], :memcpy #object[tech.v3.datatype.ffi.mmodel.G__9866$invoker_memcpy 0x15382815 "tech.v3.datatype.ffi.mmodel.G__9866$invoker_memcpy@15382815"]}}]
We just generated JDK-16 bindings and loaded our library. We can of course now memset and memcpy all day long:
user=> (def nbuf (dtype/make-container :native-heap :float32 (range 10)))
#'user/nbuf
user=> Mar 22, 2021 1:22:39 PM clojure.tools.logging$eval481$fn__484 invoke
INFO: Reference thread starting
nbuf
#native-buffer@0x00007FF8DD860270<float32>[10]
[0.000, 1.000, 2.000, 3.000, 4.000, 5.000, 6.000, 7.000, 8.000, 9.000]
user=> (libc/memset nbuf 0 40)
#object[tech.v3.datatype.ffi.Pointer 0x708f4d81 "{:address 0x00007FF8DD860270 }"]
user=> nbuf
#native-buffer@0x00007FF8DD860270<float32>[10]
[0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000]
user=>
Under the hood, calling these C functions from JNA and JDK-16 is completely different, but with this system we do not need to worry at all. Simply build out your data definition for the underlying functions and the code generation takes care of the rest.
Graal Native Executable
Let's say that we would like to make a single executable that we can run with minimal outside dependencies. We go again into our dtype-next directory and activate a Graal native installation:
chrisn@chrisn-lt-01:~/dev/clj-ffi$ cd dtype-next
chrisn@chrisn-lt-01:~/dev/clj-ffi/dtype-next$ source scripts/activate-graal
--2021-03-22 13:27:00-- https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-21.0.0.2/graalvm-ce-java8-linux-amd64-21.0.0.2.tar.gz
Resolving github.com (github.com)... 140.82.112.3
Connecting to github.com (github.com)|140.82.112.3|:443... connected.
...
chrisn@chrisn-lt-01:~/dev/clj-ffi/dtype-next$ java -version
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b07)
OpenJDK 64-Bit Server VM GraalVM CE 21.0.0.2 (build 25.282-b07-jvmci-21.0-b06, mixed mode)
We add a new source path to deps.edn file where we will target our class generation system:
{:paths ["src" "generated-classes"]
:deps {cnuernber/dtype-next {:mvn/version "6.24"}
net.java.dev.jna/jna {:mvn/version "5.8.0"}}
:aliases
{:jdk-16
{:jvm-opts ["--add-modules" "jdk.incubator.foreign" "-Dforeign.restricted=permit" "--add-opens" "java.base/java.lang=ALL-UNNAMED" "-Djava.library.path=/usr/lib/x86_64-linux-gnu"]}
:depstar
{:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.0.193"}}
:ns-default hf.depstar
:aliases [:graal-compile]
:exec-fn hf.depstar/uberjar
:exec-args {:group-id "mygroup"
:artifact-id "libc"
:version "1.00-beta-1"
:sync-pom true
:aot true
:compile-ns [graal-main]
:main-class graal-main
:jar "libc.jar"
;;Disable tensor code generation and ensure direct linking.
:jvm-opts ["-Dtech.v3.datatype.graal-native=true"
"-Dclojure.compiler.direct-linking=true"]}}}}
Next we generate the GraalVM static bindings required to tell graal native where to find the required headers what functions to generate:
(ns make-graal
(:require [libc :as libc]
[tech.v3.datatype.ffi.graalvm :as graalvm]))
(defn make-bindings
[]
(.mkdir (java.io.File. "generated-classes"))
(with-bindings {#'*compile-path* "generated-classes"}
(graalvm/define-library
libc/fn-defs
nil
{:classname 'libc.GraalBindings
:headers ["<string.h>"]
:instantiate? true})))
(make-bindings)
After compiling the above file we should be able to check out generated-classes directory and find our nice newly generated class files:
chrisn@chrisn-lt-01:~/dev/clj-ffi$ ls -1 generated-classes/libc/
'GraalBindings$directives.class'
'GraalBindings$inner.class'
'GraalBindings$invoker_memcpy.class'
'GraalBindings$invoker_memset.class'
GraalBindings.class
Next up we build out graal_main.clj:
(ns graal-main
(:require [libc :as libc]
[tech.v3.datatype.ffi :as dt-ffi]
[tech.v3.datatype :as dtype]
;;required for generated class to work correctly
[tech.v3.datatype.ffi.graalvm-runtime])
(:import [libc GraalBindings])
(:gen-class))
(defn -main
[& args]
(let [libinst (libc.GraalBindings.)
_ (dt-ffi/library-singleton-set-instance! libc/lib libinst)
nbuf (dtype/make-container :native-heap :float32 (range 10))]
(println "before memset" nbuf)
(libc/memset nbuf 0 40)
(println "after memset" nbuf)
0))
And finally we write a simple compile script that will generate our uberjar and then make our executable. Write this data to scripts/compile:
#!/bin/bash
set -e
rm -rf classes && mkdir classes
clojure -X:depstar
$GRAALVM_HOME/bin/native-image \
--report-unsupported-elements-at-runtime \
--initialize-at-build-time \
--no-fallback \
--no-server \
-H:+ReportExceptionStackTraces \
-J-Dclojure.spec.skip-macros=true \
-J-Dclojure.compiler.direct-linking=true \
-J-Dtech.v3.datatype.graal-native=true \
-jar libc.jar graal_main
We then run our compile script and execute the result:
chrisn@chrisn-lt-01:~/dev/clj-ffi$ bash scripts/compile
[main] WARN hf.depstar.uberjar - :group-id should probably be a reverse domain name, not just mygroup
[main] INFO hf.depstar.uberjar - Synchronizing pom.xml
Skipping paths: generated-classes
[main] INFO hf.depstar.uberjar - Compiling graal-main ...
[main] INFO hf.depstar.uberjar - Building uber jar: libc.jar
[main] INFO hf.depstar.uberjar - Processing pom.xml for {mygroup/libc {:mvn/version "1.00-beta-1"}}
[graal_main:32133] classlist: 4,188.88 ms, 2.07 GB
[graal_main:32133] (cap): 578.59 ms, 2.07 GB
[graal_main:32133] setup: 2,050.02 ms, 2.07 GB
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of tech.v3.datatype.UnsafeUtil. Detailed failure reason(s): Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
[graal_main:32133] (clinit): 211.62 ms, 2.81 GB
[graal_main:32133] (typeflow): 6,649.25 ms, 2.81 GB
[graal_main:32133] (objects): 4,204.02 ms, 2.81 GB
[graal_main:32133] (features): 252.99 ms, 2.81 GB
[graal_main:32133] analysis: 11,865.41 ms, 2.81 GB
[graal_main:32133] universe: 538.17 ms, 2.81 GB
[graal_main:32133] (parse): 1,333.60 ms, 2.81 GB
[graal_main:32133] (inline): 1,370.26 ms, 2.81 GB
[graal_main:32133] (compile): 13,370.47 ms, 3.17 GB
[graal_main:32133] compile: 16,769.55 ms, 3.17 GB
[graal_main:32133] image: 1,435.23 ms, 3.17 GB
[graal_main:32133] write: 256.68 ms, 3.17 GB
[graal_main:32133] [total]: 37,268.84 ms, 3.17 GB
chrisn@chrisn-lt-01:~/dev/clj-ffi$ ./graal_main
Mar 22, 2021 2:01:39 PM clojure.tools.logging$eval1$fn__4 invoke
INFO: Reference thread starting
before memset #native-buffer@0x00005632821CAAC0<float32>[10]
[0.000, 1.000, 2.000, 3.000, 4.000, 5.000, 6.000, 7.000, 8.000, 9.000]
after memset #native-buffer@0x00005632821CAAC0<float32>[10]
[0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000]
A Truly 'Next Generation' FFI for the JVM
There are a few more features we haven't talked about, such as how to deal with C structs and how to build a shared library from our code using the Graal Native compiler. These are all possible, but out of the scope of this post. What we have shown, is how to write code that uses native functions and have that code work across JNA, JDK16 and GraalVM (3 different FFI implementations!). For binding native code into the JVM, this is a truly next-generation system!
You can see a more involved example at my example avclj library. This is a simple library that allows users to encode video using a variety of codecs and into a variety of file formats. Of course, it works across all three of JNA, JKD16, and Graal - and additionally includes a shared-library compilation pathway.
We hoped you enjoyed seeing a glimpse of what is coming in terms of JVM tools and technologies. This stuff is absolutely cutting edge for the JVM and it enables us to very quickly leverage systems using a language interface that has been around as long as Unix - the C ABI - for inter-language communication. Maybe this article will convince you to spend some time exploring some newer technologies that are present in the JVM ecosystem or perhaps enable you to embed some of your tech into language ecosystems that aren't JVM based. You can find the source code behind this example in the examples folder of dtype-next.
TechAscent: Cutting clear pathways through rapidly growing technical jungles.