How CVM locates native methods (JNI, KNI, and CNI)

JNI native methods are usually placed in a shared library. On Linux, the file name for the JNI shared library is of the form lib.so. For example, libawtjpeg.so is the shared library that implements the JPEG support for AWT.

During execution of a Java program, the first time an attempt is made to call a JNI native method, CVM must find the C function that implements the native method. Every loaded method in CVM is represented by a CVMMethodBlock, and this CVMMethodBlock must be made to point to the location of the JNI native method so it can be called. How CVM locates this native method is dependent on how the Class for the JNI method is loaded. Typically there are two ways it is loaded: one for classes on the bootclaspath (-Xbootclasspath), and another for classes loaded by a ClassLoader instance (usually a class on the java.class.path classpath).

Loading native libraries for classes loaded by a ClassLoader Instance

For classes loaded by a ClassLoader instance, the Java program must first call ClassLoader.loadLibrary() to load the JNI shared library. This must be done before any attempt to is made to call any of the JNI native methods in the shared library. Usually System.loadLibrary() is used, and it will indirectly call ClassLoader.loadLibrary(), using the currently executing Class to find the proper ClassLoader instance. This is normally done from the static initializer of a Class that requires the JNI shared library. A typical call to System.loadLibrary() usually looks like this:

public Class LibraryTest
    static {
        System.loadLibrary("test");
    }
}

This will cause libtest.so to be loaded the first time a reference is made to the LibraryTest class. At this point, symbols in the shared library are available to CVM when it needs to resolve the address of native method that is being called for the first time.

Different classloaders have different ways of locating JNI shared libraries. Classes found on the classpath (-cp, -classpath, and -Djava.class.path options) are loaded using what is known as the System Classloader. For the System Classloader, the JNI shared libraries are located using the java.library.path property, so you must specify the directories of all your application shared libraries using the -Djava.class.path option. CVM will not automatically find them in the current directory or in the Unix LD_LIBRARY_PATH.

Loading native libraries for classes loaded of the bootclasspath

Core CDC classes are usually loaded via the bootclasspath. It defaults to the jar files in the ../lib directory relative to the cvm binary. Usually this is where you will find your profile classes, and you will see jar files in this directory with names like cdc.jar, foundation.jar, midp.jar, etc. There is a pseudo classloader to handle loading these classes. It is usually referred to as the Null classloader or bootclassloader. However, there is no ClassLoader instance representing it.

JNI methods for classes on the bootclasspath can either be linked with the cvm binary, or registered using a doPrivileged() call. The following shows how the libawtjpeg.so library is registered:

       java.security.AccessController.doPrivileged(
           new sun.security.action.LoadLibraryAction("awtjpeg"));

On Linux this loads libawtjpeg.so so symbols can be looked up in it. Libraries loaded in this manner must all be located in one of the directories specified by the sun.boot.library.path property, which by default includes the ../lib directory. When a reference is made to a JNI method in the bootclasspath, CVM looks for it in the cvm binary and also in the shared libraries registered as indicated above.

JNI methods for romized classes

When a class is romized, a C file is generated that contains the runtime representation of the class. This C file is compiled and statically linked with the cvm binary. This is done rather than dynamically loading the class at runtime. There is a core set of CDC classes that are always romized. CVM will not build or run otherwise. All other library classes can by dynamically loaded or romized. They are romized when CVM is built with CVM_PRELOAD_LIB=true.

As mentioned earlier, for each native method there is a field in the CVMMethodBlock that points to the native message. For romized methods this reference needs to be resolved by the linker at build time. For this reason JNI methods for romized classes are statically linked with the cvm binary, and there is no need for a lookup of these methods to be done at runtime. If you ever see an linker error message for an unresolved JNI method, it's probably because it's a method in a romized class, and the native method was declared but never implemented, or possibly the C name of the native method was done incorrectly.

For romized builds, you won't find libraries like libawtjpeg.so in the ../lib directory since instead of building a shared library, at build time the JNI methods are just statically linked with the cvm binary. In this case the LoadLibraryAction() call like in the example above will end up throwing an exception, and you don't want this. For this reason, at build time you need to give CVM a list of libraries to be treated as "internal" or built-in, so it can ignore LoadLibraryAction() calls made for them. For libawtjpeg.so, the following is done in the makefiles:

AWT_LIB_NAME    ?= $(AWT_IMPLEMENTATION)awt
JPEG_LIB_NAME    ?= awtjpeg
ifeq ($(CVM_STATICLINK_LIBS), true)
BUILTIN_LIBS += $(AWT_LIB_NAME) $(JPEG_LIB_NAME)
endif

Note that by default CVM_STATICLINK_LIBS=$(CVM_PRELOAD_LIB), although it is possible to set them differently.

See the next section for special considerations when romizing CNI and KNI methods.

CNI and KNI methods

CVM also supports another native method variant called CNI. It's original purpose was to provide a very lightweight and fast native interface, so it is used to get performance gains with small methods that are called frequently. It is also used for native methods that need access to VM internals, and these CNI methods are viewed as extensions of the VM. More recently CNI methods have been used to support using KNI on CVM. All KNI methods in CVM are actually CNI methods. Thus platform developers are more apt to implement CNI methods now.

Since JNI and CNI methods both use different C naming conventions, the romizer (JCC) needs to know which type of native methods a class implements so it can generate the reference to the native methods with the proper symbol name. Otherwise a link error will result at build time because the reference to the native method and the actual C name of the native method will not match. This is handled in the makefiles by adding the class to the list of CNI classes:

    CVM_CNI_CLASSES += MyKNIClass

Note that because of this, a romized class must choose to implement its native methods as either CNI or JNI, but can't mix in both.

For dynamically loaded classes with CNI methods, just like with JNI methods, the first attempt to call a native method will cause CVM to look up the address of the method. It will first lookup the native method using its JNI name. If that fails it will then attempt with the CNI name. For this reason the build system does not need to know which type of native methods a class has if the class will be dynamically loaded. It also allows a dynamically loaded class to have a mix of both JNI and CNI methods.

Debug versions of native libraries

When CVM is built using CVM_DEBUG=true, it expects native libraries to contain the _g suffix. For example, libawtjpeg_g.so. For native libraries built along with CVM, this is handled automatically by the makefiles. If you are using your own native library and expect to run debug versions of CVM, it is best to just always build debug and non-debug versions of the native library. Otherwise you'll get an unexpected UnsatisfiedLinkError when switching between debug and non-debug versions of CVM (see next section). If you are given a non-debug native library and want to use it with a debug version of cvm, just rename it or copy to a file whose name has the _g suffix.

UnsatisfiedLinkError: Debugging native library and method lookup failures

If CVM fails to find or load your native library, you'll see an exception like the following:

java.lang.UnsatisfiedLinkError: /opt/cdc/lib/libqtawt.so
   at java.lang.ClassLoader$NativeLibrary.load(Native Method)
   at java.lang.ClassLoader.loadLibrary0(Unknown Source)
   at java.lang.ClassLoader.loadLibrary0(Unknown Source)
   at java.lang.ClassLoader.loadLibraryInternal(Unknown Source)
   at java.lang.ClassLoader.loadLibrary(Unknown Source)
   at java.lang.Runtime.loadLibrary0(Unknown Source)
   at java.lang.System.loadLibrary(Unknown Source)
   at sun.security.action.LoadLibraryAction.run(Unknown Source)
   at java.security.AccessController.doPrivileged(Unknown Source)
   at java.security.AccessController.doPrivileged(Unknown Source)
   at java.awt.QtGraphicsEnvironment.<init>(Unknown Source)
   at java.awt.QtToolkit.<init>(Unknown Source)
   at java.lang.reflect.Constructor.newInstance(Unknown Source)
   at java.lang.Class.newInstance(Unknown Source)
   at java.awt.Toolkit$1.run(Unknown Source)
   at java.security.AccessController.doPrivileged(Unknown Source)
   at java.security.AccessController.doPrivileged(Unknown Source)
   at java.awt.Toolkit.getDefaultToolkit(Unknown Source)
   at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(Unknown Source)
   at java.awt.Window.<init>(Unknown Source)
   at java.awt.Frame.<init>(Unknown Source)
   at java.awt.Frame.<init>(Unknown Source)
   at basis.DemoFrame.<init>(Unknown Source)
   at basis.DemoFrame.main(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at sun.misc.CVM.runMain(Unknown Source)

Here are some possible causes:

• The library was named incorrectly, such as when you run a debug version of CVM and it expects the _g suffix.
• The library was found but failed to load because it is meant for another platform.
• The library was found but failed to load because the library itself has unresolved symbols in it that need to be resolved by another shared library. A common case of this is when the native library links against some other shared library. That secondary library must be automatically located by the loader (using LD_LIBRARY_PATH for unix). It will not be found by including it in java.library.path or sun.boot.library.path since it not the responsibility of cvm to load it.

If you run a debug version of CVM, for some failures you will see some extra information at the start of the exception that may help explain why the library failed to load. The following are examples of the types of information you may see reported with the UnsatisfiedLinkError. The java backtraces have been ommited. In the first two examples, the first line of output is only present for debug builds:

Example #1: libqte.so.2 is not in LD_LIBRARY_PATH, but is needed by libqtawt_g.so:

libqte.so.2: cannot open shared object file: No such file or directory
java.lang.UnsatisfiedLinkError: /opt/cdc/lib/libqtawt_g.so

Example #2: libqtawt_g.so is not a valid shared library for this platform:

/ws/phoneme/trunk/cdc/build/linux-x86-generic/lib/libqtawt_g.so: invalid ELF header
java.lang.UnsatisfiedLinkError: /opt/cdc.lib/libqtawt_g.so

Example #3: libqtawt_g.so is not found:

java.lang.UnsatisfiedLinkError: no qtawt in sun.boot.library.path

Example #4: cvm fails to find the symbol for MyClass.myNativeMethod()V in any registered library:

java.lang.UnsatisfiedLinkError: MyClass.myNativeMethod()V
       at MyClass.main(Unknown Source)
       at java.lang.reflect.Method.invoke(Unknown Source)
       at sun.misc.CVM.runMain(Unknown Source) 

If you run into an error like example #4, make sure that the native library was successfully loaded. If you did not get a previous UnsatisfiedLinkError as mentioned above (and there is no code in place to catch and hide an UnsatisfiedLinkError), then it can be assumed it was loaded ok. Most likely either you forgot to implement the native method or misnamed it. Using nm to look at the symbols in the native library is a good way to start to make sure the symbols are being defined correctly.

For most ports (the most notable exception being Windows) the code that does the symbol lookup is CVMdynlinkSym() in src/portlibs/dlfcn/linker_md.c. You might want to try adding some debug code there to get more error information. For starters, printf() the symbol name it is trying to lookup when it fails, and make sure it matches the name of your native method in your C source file.

-- ChrisPlummer - 08 Feb 2007