Crash when bootimage runner uses longjmp

Description

Reported by Carl Ritson on the researchers mailing list [1]:

In bootImageRunner/sys.C, the function sysThreadTerminate() uses
longjmp() to return from the VM stack to the thread stack at thread
termination.

In recent versions of glibc, longjmp() passes through a checking
function __longjmp_chk() which attempts to ensure that the jump is to
a valid stack frame. It does this by asserting that the new stack
pointer is either greater than then current one, or contained on the
alternate signal stack. As stacks grow downward (on x86 at least),
this ensures that longjmp() only ever unwinds the stack. This
assertion is obviously flawed when an separately allocated stack is
used, such as in JikesRVM.

On a modern Linux kernel mmap() allocates memory top-down, so
successive allocations (tend to) fall on lower addresses. Thus in
practice thread stacks are higher in memory than VM stacks, so the
__longjmp_chk() assertion succeeds. However, a legacy mode of
bottom-up allocation is also supported, this can be activated by
setting the process personality (e.g. setarch -L) or having an
unlimited stack size in the process resource limits (e.g. ulimit -s
unlimited). The second of these is obscure, but it what resulted in
my discovery of this issue.

In bottom-up mode thread stacks are allocated below the VM memory area
(at least on x86_64 32-bit mode), and hence __longjmp_chk fails.
For JikesRVM in production configuration on x86_64-linux the error
looks like the following (produced from DaCapo 2006 eclipse, on Ubuntu
12.04 LTS):

There's also a bootloader crash that occurs on an unmodified PPC32 system which seems to have the same cause (no stacktrace is provided in that case).

[1] http://sourceforge.net/mailarchive/message.php?msg_id=30708025

Environment

IA32 and PPC (confirmed for x86_64 and ppc32)

Status

Assignee

Erik Brangs

Reporter

Erik Brangs

Labels

None

Fix versions

Affects versions

Priority

Low
Configure