Reported by Carl Ritson on the researchers mailing list :
In bootImageRunner/sys.C, the function sysThreadTerminate() uses
longjmp() to return from the VM stack to the thread stack at thread
In recent versions of glibc, longjmp() passes through a checking
function __longjmp_chk() which attempts to ensure that the jump is to
a valid stack frame. It does this by asserting that the new stack
pointer is either greater than then current one, or contained on the
alternate signal stack. As stacks grow downward (on x86 at least),
this ensures that longjmp() only ever unwinds the stack. This
assertion is obviously flawed when an separately allocated stack is
used, such as in JikesRVM.
On a modern Linux kernel mmap() allocates memory top-down, so
successive allocations (tend to) fall on lower addresses. Thus in
practice thread stacks are higher in memory than VM stacks, so the
__longjmp_chk() assertion succeeds. However, a legacy mode of
bottom-up allocation is also supported, this can be activated by
setting the process personality (e.g. setarch -L) or having an
unlimited stack size in the process resource limits (e.g. ulimit -s
unlimited). The second of these is obscure, but it what resulted in
my discovery of this issue.
In bottom-up mode thread stacks are allocated below the VM memory area
(at least on x86_64 32-bit mode), and hence __longjmp_chk fails.
For JikesRVM in production configuration on x86_64-linux the error
looks like the following (produced from DaCapo 2006 eclipse, on Ubuntu
There's also a bootloader crash that occurs on an unmodified PPC32 system which seems to have the same cause (no stacktrace is provided in that case).