Because I keep getting this wrong, here is where the pointers in an interpreterState should point. This is a PPC32 frame with a four-word stack with one item pushed onto it and no monitors:

…680 Back chain
…684 LR save word
…688 Padding
…68c
…690 Slop
…694 ← _stack_limit
…698  
…69c Stack
…6a0 ← _stack
…6a4  
…6a8 Interpreter state ← _stack_base, _monitor_base
.
.
.
…6ec

This is the same frame with a monitor allocated:

…680 Back chain
…684 LR save word
…688 Slop
…68c ← _stack_limit
…690  
…694 Stack
…698 ← _stack
…69c  
…6a0 Monitor ← _stack_base
…6a4
…6a8 Interpreter state ← _monitor_base
.
.
.
…6ec

My landline went dead shortly after starting work today. I managed to keep coding for an hour or so but my PPC machine is in Boston and there’s only so much you can do blind. It turned out to be a blessing in disguise however, since I’ve been meaning to document the interpreter calling convention for some time, but I got bored every time and went back to coding. Until today…

So, the interpreter calling convention. The interpreter calling convention is basically the register usage and the stack frame layout used within the interpreter — which for the C++ interpreter means everything inbetween StubRoutines::call_stub() and BytecodeInterpreter::run(). And while not immediately relevent, this includes any code created by a JIT.

The register usage is the simplest bit. There are three symbolically named registers that are valid at all times within the interpreter:

Rmethod
The address of the current methodOop.
Rlocals
The address of first local variable. Local variables are accessed with negative indices, so the address of the second local variable is Rlocals - wordSize and so on. Note that Rlocals is essentially a stack pointer and is treated as such in the result convertors, so you need to be careful it’s pointing where you expect while methods are returning.
Rstate
The address of the current interpreterState object.

These registers are assigned to registers defined as non-volatile by the PPC ABIs, so they do not need to be saved or restored around calls to native ABI functions.

The stack is managed in accordance with the PPC ABIs, with r1 as the frame pointer and with frames laid out in the standard manner: 

    | ...                  |  high addresses
+-> | Link area            |
|   +----------------------+
|   | Register save area   |
|   | Local variable space |
|   | Parameter list space |
+---+ Link area            |  low addresses
    +----------------------+

Each method has its own frame, the “local variable space” of which is laid out as follows: 

    +----------------------+
    | interpreterState     |  high addresses
    +----------------------+
    | monitor 0            |
    |  ...                 |
    | monitor m            |
    +----------------------+
    | stack slot 0         |
    |  ...                 |
    | stack slot n         |
    +----------------------+
    | slop_factor          |
    +----------------------+
    | padding              |  low addresses
    +----------------------+

slop_factor is a hack. When a method is called the callee’s parameters are pushed onto the caller’s stack. The method pops these off and pushes its return value, if any. It is a given that the Java compiler allocates enough stack slots for the parameters a method will call, but nobody seems sure it allocates enough slots for a return value in the event that the return value requires more slots than the parameters. This is dubiously referred to as “the static long no_params() issue”. slop_factor is essentially two extra words above the expression stack to protect what follows from being overwritten in this case. It’s probably unnecessary, but on PPC32 there is a fair chance that a stack overrun will corrupt the return address, just the kind of fun bug where the crash happens long after the cause. I left it in; I value my sanity over some wasted bytes of memory.

The state-monitors-stack ordering is not random. On entry to a method the caller’s frame will be at the top of the stack and Rlocals will be pointing at an expression stack slot within that frame (or at the higher slop word, in frames with no stack). A method’s N parameters are its first N local variables, and in methods with more local variables than parameters this frame (the caller’s frame) may be extended to accomodate them. Having the expression stack the lowest thing in the frame minimises the amount of stuff that needs moving when a frame is extended. Of course, this means that the entire expression stack needs moving every time a monitor is allocated, but synchronized methods pre-allocate a monitor so the monitor list needs extending much less frequently than the expression stack. Having it this way round means the monitors never move — and you can’t move a monitor without a safepoint so this is nice.

I’m fading now, but one final point is that random frame extension means you cannot unwind frames assuming they’re the same size they were when you created them.

Being at a point where I’m interpreting bytecodes is really cool: every new thing I implement gets the interpreter a whole load further. I’ve now executed 70 instructions, including calls to both native and non-native methods with both void and non-void return values. It’s currently stopping at the start of the first method to require more locals than it takes parameters: the first method whose frame may need expanding, in other words. This might be slightly complicated by the fact that Rlocals is not used exactly as I thought it would be when I planned the frame-expanding code. I’m hoping this doesn’t matter.

I just interpreted a bytecode:

$ control/build/linux-ppc/bin/java -XX:+TraceBytecodes
VM option '+TraceBytecodes'

[12474] static void java.lang.Object.<clinit>()
[12474]        1     0  invokestatic 0 <registerNatives> <()V>
[12474]        2     3  return
error: bad message from interpreter: 9

A couple of people have asked me to explain what I mean by “the C++ interpreter” and why bits of it are written in assembler. Here is a brief summary:

  • OpenJDK is the implementation of the Java platform of Sun Microsystems.
  • Hotspot is the Java virtual machine of OpenJDK.
  • Hotspot comprises two interpreters and two JITs.
  • A running instance of Hotspot comprises one interpreter optionally supplemented by one JIT.
  • The interpreters are the template interpreter (aka the asm interpreter) and the C++ interpreter (aka the bytecode interpreter).
  • The JITs are client (aka compiler 1 aka C1) and server (aka compiler 2 aka C2).

The point of intersection between the two interpreters and the two JITs is method dispatch. Every method in Hotspot is defined by a methodOop, and each methodOop has a method entry* which is the address of the code that will execute the method. Different methods have different entries, depending upon whether they are native or not, synchronized or not, JIT-compiled or not, etc etc, but every method is invoked the same way: you put the address of the methodOop in one register, the address of the parameters in another, then jump to the method entry. The way the C++ interpreter differs from the template interpreter is that in the template interpreter everything is done in assembler whereas in the C++ interpreter the non-native entries simply** call BytecodeInterpreter::run(). The C++ interpreter’s entries are therefore much smaller and so easier to port. But they are still written in assembler.

* This is a slight lie: they have two.
** ha ha ha ha ha!

I think I’ve finally settled on the basics of an interpreter calling convention. It’s difficult as I don’t really know what I’ll need in the interpreter and it took a bit of thrashing around while I tried to figure out what to do with the stack, but here goes:

  1. There are two non-volatile registers:

    Rmethod is the address of the current methodOop
    Rlocals is the address of first local variable

    These are expected to be valid at all times within the interpreter.

  2. In addition there are two volatile registers:

    Rnparams is the number of parameters
    Rnlocals is the number of local variables (including parameters)

    These are only expected to be valid at interpreter entry points. They don’t even need to be in registers (you can read them from the methodOop) but a) they are already in registers in call_stub and b) both are needed in registers in the entry points, so as long as it’s no trouble to pass them like this I will continue to do so.

  3. The stack frames will be laid out in accordance to the PPC ABIs: 
        | ...                  |  high addresses
+-> | Link area            |
|   +----------------------+
|   | Register save area   |
|   | Local variable space |
|   | Parameter list space |
+---+ Link area            |  low addresses
    +----------------------+
  4. The area referred to by the ABIs as “local variable space” will be arranged as follows: 
        [local variable Rnlocals]
      ...
    [local variable 1       ] <-- Rlocals
    [padding as required    ]

    Such that the first local variable is accessed as 0(Rlocals), the second as wordSize(Rlocals), and so on. This only works if always know in advance how many local variables the method we are calling will need, which seems reasonable. If there are cases where this isn’t so I can insert a check in the method entry to resize the frame as necessary, but this is expected to be time-consuming so should be the exception rather than the rule.

  5. Any additional stack slots will be allocated below the first local variable, such that the first additional stack slot will be referenced as -wordSize(Rlocals).
  6. Monitors will be allocated below any additional stack slots. I may well always allocate some space for monitors depending on how frequently they are created, how many any given method is expected to require, and exactly how time-consuming a frame-resize is.

Ok, I think that’s it.

This stack thing is proving to be a real pig. The problem is that the stack for a method is set up in two parts. The caller allocates space for the method’s parameters and fills them in prior to calling the method entry, and the method entry then extends the stack to allow for any additional local variables. This is fine on i486 and amd64 (and I think sparc too) where you have free reign to set the stack pointer to whatever you want, but an absolute pain on ppc where the ABI dictates the stack be arranged in frames. (I think this means that the “stack pointer” on ppc is actually a frame pointer.) I thought I could get around this because the caller can look in the methodOop to see how many additional local variables are required; you’d just allocate them at the same time as the parameters (which are just the first however many local variables anyway). Except it turns out that the C++ interpreter can arbitrarily extend the stack too, to allocate space for monitors, so I still need to figure out a way to do it!

aph pointed out that signals make a mess of my idea of writing below r1 and protecting later, but I realised that you know in advance how much stack a method will use so you can just set up the stack for that before you call it. Of course, all that has to be in assembler, so I can’t use my funky StackFrame class :( I decided it’s high time I made a table of register usage across ABIs to aid me in writing this:

  ppc ppc64
r0 Volatile register which may be modified during function linkage
r1 Stack frame pointer
r2 Reserved TOC pointer
r3 Volatile registers used for parameter-passing and return values
r4    
r5-r10 Volatile registers used for parameter-passing
r11 Volatile registers which may be modified during function linkage Volatile register used in calls by pointer and/or as an environment pointer
r12 Volatile register used in function linkage and exception handling
r13 Small data area pointer System thread ID
r14-r30 Non-volatile registers used for local variables
r31 Non-volatile register used for local variables or as an environment pointer