Zero and Shark in IcedTea

Over the past few months I’ve been working on Shark in it’s own forest. This has allowed me to track upstream HotSpot (and the goal is to upstream Shark, so it’s the correct place to base it) but it’s meant that the Shark (and Zero) in IcedTea6 are old. I’m trying to update Zero and Shark in Icedtea6, but it’s a nightmare.

Zero in IcedTea6 has the ARM interpreter which can’t go upstream. Upstream has all the JSR 292 stuff which can’t go downstream. Between these two are fixes that need synchronizing, in both directions. One of the fixes, 6939182 (aka PR icedtea/323) requires the ARM interpreter to be updated before it can be committed, so that needs keeping separate… except that the changes to Shark to support that one are pretty invasive and hard to strip out.

It’s a real mess. I’m pretty close to giving up on it.

New stack overflow code

It’s funny, but I’d kind of thought that the version of Zero that got upstreamed was pretty much cut and dried. It hadn’t had any real changes for months, after all, and the S/390 compatibility work required was so minimal—in terms of code changed, if not in terms of time taken!—that I didn’t think there was much more to do. But in this last couple of months I’ve made a ton of fixes, including one big change that’s upstream (improved stack overflow handling) and one that’s hopefully heading that way (some frame anchor fixes).

The new stack overflow code is pretty awesome (if I say so myself!) The old code jumped through some pretty bizarre hoops in order to throw the exception from the exact frame of the method being called. This is more or less how it happens in “classic” HotSpot, which decides which method you’re in from the ABI stack, but wasn’t workable for Zero which might have run out of Zero stack with no room to put the frame of the method from which you want to throw the exception. Argh! So I wrote this look-ahead detection thing, to throw the exception when you still had some stack to put your frame on, which worked, but was easily fooled. A valid Java method could have a frame far larger than the default thread stack size on most systems, so the lookahead worked on the assumption that a frame probably wouldn’t be that big. There were a number of places where the stack could overflow, in which case you’d hit an Unimplemented and the VM would bomb out. Not good, and not very secure either.

The new code is awesome. I realised that it doesn’t really matter exactly which frame the StackOverflowError got thrown from, only that it got thrown. I wrote a little method, ZeroStack::overflow_check, which is inline and can be called from anywhere—any VM state, etc. It does the check, and throws the exception, via ZeroStack::handle_overflow. That one is pretty complicated, but I did say I wanted the overflow check to be callable from anywhere, so it has to set up the frame anchor if it isn’t already, and cope with whatever thread state it finds itself in. The best thing about ZeroStack::handle_overflow? You can call it from Shark, which let me blow away all the ugly stack overflow code I was griping about in my last post.

Those Unimplementeds in the stack overflow detector were my least favourite thing about Zero, my guilty secret (in as much as open source code published on the internet is secret). And now they’re gone! No more ugly code in Zero! Until a few days later, when I discovered some stuff in the frame anchor that was reversed. But that’s for another day.

JNI wrapper compilation

I now have a version of Shark with a basic implementation JNI wrapper compilation. Sadly I can’t say if it’s faster or not yet because it’s totally unstable!

The problem is this. When HotSpot wishes to compile a normal (interpreted) method, the thread initiating the compile simply adds it to a queue and carries on doing whatever it was it was doing. A separate thread, the compiler thread, loops over this queue, compiling methods one at a time. This means there’s only ever the one thread making LLVM calls, and everything is rosy.

When HotSpot wishes to compile a native (JNI) method, the thread initiating the compile bypasses the queue and does it immediately. It acquires a lock, the adapter hander library lock, so there’s only every one native method being compiled at once, but in the meantime the compiler thread is in all likelihood busy compiling some normal method or another, so there’s two threads making LLVM calls. LLVM doesn’t like this, not LLVM prior to 2.6 at any rate, and even then not without being written with a separate LLVMContext for each thread.

The obvious fix for this is for Shark to acquire a lock before compiling either a normal or a native method, ensuring that only one thread is calling into LLVM at once. This doesn’t work, however, as the compiler thread runs _thread_in_native. The benefit of this is that the compiler thread does not have to halt for safepoints (and the rest of the VM doesn’t have to wait for the compiler thread to halt) but the drawback of this is that threads running _thread_in_native may not own locks. You can’t make the compiler thread run other than _thread_in_native, not without losing the large chunk of the server compiler that Shark shares, and you can’t hack a lock in there anyway (by using a pthread mutex, say, rather than a HotSpot lock) because it’ll deadlock the first time a safepoint occurs when the compiler thread holds the lock (compiling a normal method) and a Java thread is blocking trying to take it (to compile a native one).

I’ve been circling around this issue for a couple of days now, but it looks like the only solution is to require LLVM 2.6 and rearrange everything to use separate contexts.

Long overdue update

It’s been a while; time for a catchup!

June and July I mostly spent cleaning up Shark. HotSpot’s existing JITs, client and server, both inline pointers to objects in the native code they generate. These pointers need to be visible to the garbage collector, both so it knows the objects are live and so it can rewrite the pointers if it moves the object. This is trivial for client and server, as they both have access to the native code they generate: each method’s code is accompanied by a list of pointer locations within it. Shark, on the other hand, has no access to its generated native code other than knowing its address and size. Pointers can’t be inlined in Shark — it can’t tell the garbage collector where they are — so Shark had to load all garbage collected object pointers from other places, generally wherever the interpreter stored them.

This caused no end of problems. Aside from requiring more loads than the other JITs (at least one per object, and sometimes three or four) Shark had to mirror huge chunks of the interpreter. It had to cope with objects that were loaded in the VM (so the compiler could see them) but not cached in the interpreter (where the compiled code could see them). Because Shark wasn’t behaving like the other compilers, HotSpot’s compiler support layer would break in all kinds of exciting and imaginative ways. Finally, the method used by the server JIT to optimize interface calls to virtual calls and virtual calls to direct calls could not be used. Aside from the obvious speedup, Shark can only inline direct calls, so reducing virtual and interface calls to direct calls exposes them to the inliner. Calls in Shark have a lot of overhead, so this would have been a big win.

Sometime in May I figured out how to fake inlined object pointers. HotSpot’s compiler interface expects the compiler to generate native code into a CodeBuffer. Shark, of course, uses LLVM, which generates code into a buffer it allocates. Shark had a HotSpot code buffer, but it didn’t do a lot with it. Now, every time Shark has an object pointer to inline, it writes it into the HotSpot code buffer where the garbage collector can see it. The generated code then loads the object pointer from the code buffer whenever it needs it. The pointer is still not inlined — there’s still a load required — but now it’s always only one load. Not a big speedup in itself, but it meant the remaining interpreterisms could be removed, which fixed the support layer breakages and allowed me to copy the interface-virtual-direct call optimization code more or less directly from the server compiler. Everything got a lot more stable, a lot more clean, and a little bit faster in the bargain.

During August I began the (long!) process of preparing Zero for submission to OpenJDK proper. It took some time to get started, but the patch has now gone through a couple of cycles of being reviewed by the HotSpot team: the code has been reformatted, the build system has been almost completely rewritten, and a bunch of other things got changed. It’s still ongoing, but the HotSpot part of the patch seems close to acceptance and the much smaller remainder will hopefully be reviewed soon. I’ve been ramping up my testing with each step: this one bootstrapped and built itself on 32-bit x86, x86_64 and 32-bit PowerPC, and has bootstrapped itself and is in the process of building itself on 64-bit PowerPC and 64-bit zSeries.

Also in August, Ed Nevill released his assembler interpreter for ARM. It replaces part of Zero with hand-crafted assembly language, making OpenJDK 2-8 times faster on that platform.

After the Zero patch is accepted, my next task will be getting Zero certified on 64-bit zSeries. I won’t have a lot of time for Shark until that’s done, but I have one last thing I want to do before I step aside for a couple of months. Xerxes Rånby posted some benchmarks of Zero, Shark, and the assembler interpreter on ARM; Shark is gratifyingly faster than everything on five of the tests, but considerably slower than the assembler interpreter on the other four. I’m not happy with that!

On the tests where it’s slower, Shark is showing very little improvement over Zero, which suggests that these benchmarks are not spending a lot of time interpreting bytecode (which Shark would have compiled and made faster). I suspect these benchmarks are spending a lot of time in JNI calls. Back in February, Ed Nevill posted some profilies he had made to figure out why some interpreter improvements he had made had had very little effect; those profiles seemed to imply that the VM was spending a lot of its time setting up JNI calls. Zero uses libffi for this, and we at Red Hat have long suspected that libffi is slow.

HotSpot’s JITs have the capability to “compile” JNI methods. This sounds odd, as JNI methods are already native code; what’s actually getting compiled is the interface between the JVM and the native JNI code. If Shark could compile JNI methods, whenever HotSpot found a hot JNI method it would be able to replace its generic, one-size-fits-all interface code (using libffi) with an LLVM-generated interface custom built specifically for that method. I’m going to spend a week or so making Shark able to compile these methods, before I descend into zSeries TCK hell…