Loading two native libraries via JNI under Linux?

Question

linux

Recently, a rare task for me appeared at work, I state it. I will say right away that I have not yet found a solution in the forehead, and I know the workarounds, but they require help from the supplier of one of the third-party libraries, and therefore the task is more than relevant.

Connoisseurs of JVM And Unix - VERY welcome ...

So, the task. There are two native libs, A and B, or, in Linux naming, these are libA.so and libB.so. A depends on B (must call functions from it). A - written by me, B - provided by third-party developers in the form of one miserable .so file, there is no access to the source.
I need to load these two libs into the JVM via JNI and call a function from library A, which in turn uses a function from B.

The following code:

static {
System.loadLibrary(B);
System.loadLibrary(A);
}

Works great in Windows (for .dll files, respectively), provided that both libraries are in java.library.path (by the way, I wrote earlier in the blog how you can change java.library.path at runtime).
How it works in Windows - B.dll is loaded, then A.dll tries to load, sees that B.dll, on which it depends, should be loaded, sees that B.dll has already been loaded into the process memory, and does not try to find and load B yourself by looking for it in PATH.

Now what seems to be happening in Linux.
Additional logging shows that libB.so is correctly located in java.library.path and loaded. Then, the runtime linker tries to load libA.so, and notices that SUDDENLY, libB.so is not in LD_LIBRARY_PATH! And I don’t want to change LD_LIBRARY_PATH, not only is it unsportsmanlike, but I want to solve problems locally. To clarify - both libraries are in a directory that is included in the java.library.path, but is NOT included in the LD_LIBRARY_PATH).

Does anyone have any ideas?
I've tried quite a few hacks already, including linking libA.so with destroying the symbol table, loading libB.so dynamically from libA.so, getting a pointer to the desired function (mangled) and calling it - none of this works the way I need.

Once again I will clarify the conditions - I do not want to modify LD_LIBRARY_PATH, I do not want to put my libs in "well-known" places like /usr/lib.
Instead of getting libB.so statically (libB.a) and hard-linking it into libA.so seems like a good option to me, but there are no sources.
Convert libB.so -> libB.a? Theoretically, it's possible.

Answer 1

Well, the native linker needs to know where to get the lib from, right? JVM is out of business here, modify LD_LIBRARY_PATH ;)

Answer 2

In the end, I solved this problem in the following way: it turned out that both libraries needed to be linked with the -Wl,soname. Since I didn't have access to the source code of the third-party library, I persuaded its creator to send me the object files, and linked it myself :)
Thank you all for your help! :)

Answer 3

This is understandable, the question is why under Windows the native linker does not try to take it at all (since it is already loaded into the memory of the JVM process), but under Unix it tries :)
Judging by the code in ClassLoader.java, at least at the java level JDK code under both Windows and Linux, links to cached libraries (private static Stack nativeLibraryContext = new Stack();) are stored in the same way anyway.

Answer 4

But isn't ld enough to link two libs into one? I don't know this tool very well, maybe someone else will tell me.
besides, why can't you run the binary with LD_PRELOAD?

Answer 5

Yes, by the way, I'm very sorry, but I had a typo there (it was a typo in the post, everything was correct in the code! :)
In the post instead of:
Actually, of course, the reverse order:
Instead
of static {
System.loadLibrary(A);
System.loadLibrary(B);
}
should be:
static {
System.loadLibrary(B);
System.loadLibrary(A);
}
And, accordingly, the loading order in the explanation below is also different. I corrected the post.

Answer 6

One more little explanation. In the code, everything was correct and so, it was a mistake that crept into the post on Habré.
libA.so is my library, libB.so is someone else's library, A is linked from parameters -L /path-to-lib-B -lB, B is loaded from Java first, then A. And this does not work under Linux (under Windows similar trick works!), logging shows the following - System.loadLibrary(B); - works and loads libB.so, then the System.loadLibrary(A) call fails with a message that the libB.so file cannot be found (although this library is already loaded into the process memory).
I myself was sure that this should work, then I was surprised.
So far, my hypothesis (you need to look at the native code in the JDK that actually loads the libraries, as I understand it is dlopen () under Linux), when loading the library libA.so dlopen does not check before trying to dynamically link library A "is B loaded into the process memory or not”, you can check this by passing the RTLD_NOLOAD flag to dlopen. It is also possible that this is the internal behavior of the Unix runtime linker, which the JVM cannot influence in any way.
One way or another - I was also sure that this should work, but for some reason it does not work.

Answer 7

I dug a little into the code, I'm interested in comments from JVM developers. Maybe I should write a separate post about it.
So, everyone knows that native libraries are loaded through the java.lang.ClassLoader class, and the transition point to native code is java.lang.ClassLoader.NativeLibrary.load.
We look at jdk/j2se/src/share/native/java/lang/ClassLoader.c, we see: We look at where the JVM_LoadLibrary function is defined, scrolling through jvm.h headers, we find something similar to what we need in the jdk/hotspot/ file src/share/vm/prims/jvm.cpp: i.e. the dll_load function is already similar to the one where a natural call to system functions for loading (POSIX / WINAPI) should go, otherwise it is already too many levels. We look at the jdk/hotspot/src/os/os/linux/vm/os_linux.cpp file: we see:
....
JNIEXPORT void JNICALL
Java_java_lang_ClassLoader_00024NativeLibrary_load
(JNIEnv *env, jobject this, jstring name)
{
const char *cname;
jint jniVersion;
jthrowable cause;
void * handle;
if (!initIDs(env))
return;
cname = JNU_GetStringPlatformChars(env, name, 0);
if (cname == 0)
return;
handle = JVM_LoadLibrary(cname);
if (handle) {
// разнообразные проверки и обрабока исключений
}
...

JVM_ENTRY_NO_ENV(void*, JVM_LoadLibrary(const char* name))
//%note jvm_ct
JVMWrapper2("JVM_LoadLibrary (%s)", name);
char ebuf[1024];
void *load_result;
{
ThreadToNativeFromVM ttnfvm(thread);
load_result = hpi::dll_load(name, ebuf, sizeof ebuf);
}
if (load_result == NULL) {
char msg[1024];
jio_snprintf(msg, sizeof msg, "%s: %s", name, ebuf);
...
Handle h_exception =
Exceptions::new_exception(thread,
vmSymbols::java_lang_UnsatisfiedLinkError(),
msg, Exceptions::unsafe_to_utf8);
THROW_HANDLE_0(h_exception);
}
// Loads .dll/.so and
// in case of error it checks if .dll/.so was built for the
// same architecture as Hotspot is running on
void * os::dll_load(const char *filename, char *ebuf, int ebuflen)
{
void * result= ::dlopen(filename, RTLD_LAZY);
if (result != NULL) {
// Successful loading
return result;
}
Ha! Those. shared objects are still loaded without the additional RTLD_NOLOAD flag! Those. JVM does not check, before loading, loaded acc. either or not yet?
For comparison, let's look at a similar Windows code, look at the os_windows.cpp file: And now look at the documentation in LoadLibrary here: msdn.microsoft.com/en-us/library/windows/desktop/ms682586.aspx If a DLL with the same module name is already loaded in memory, the system checks only for redirection and a manifest before resolving to the loaded DLL, no matter which directory it is in. The system does not search for the DLL. Those. behavior for Linux and Windows will be different in this case. Reading:
// Loads .dll/.so and
// in case of error it checks if .dll/.so was built for the
// same architecture as Hotspot is running on
void * os::dll_load(const char *name, char *ebuf, int ebuflen)
{
void * result = LoadLibrary(name);
if (result != NULL)
{
return result;
}

Answer 8

For some nefarious reason, the kernel.org mana doesn't say whether dlopen() with the RTLD_NOLOAD flag tries to find a .so file on the filesystem in the process of checking if such a library is loaded or not.
All that says:
RTLD_NOLOAD (since glibc 2.2)
Don't load the library. This can be used to test if the library is
already resident (dlopen() returns NULL if it is not, or the library's
handle if it is resident). This flag can also be used to promote the
flags on a library that is already loaded. For example, a library that
was previously loaded with RTLD_LOCAL can be reopened with
RTLD_NOLOAD | RTLD_GLOBAL. This flag is not specified in POSIX.1-2001.