Details

    • Type: Bug
    • Status: Closed
    • Priority: Highest
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0
    • Component/s: None
    • Labels:
      None

      Attachments

        Activity

        dgrove David Grove created issue -
        Hide
        dgrove David Grove added a comment -

        I looked at half a dozen; they all have this as the stacktrace:

        JikesRVM: internal error: recursive use of hardware exception registers (exiting)

        – Stack –
        at [0x684f39f8] Lorg/jikesrvm/runtime/RuntimeEntrypoints; deliverHardwareException(II)V at line 694
        at [0x684f3a0c] <hardware trap>
        at [0x684fb8c4] Lorg/jikesrvm/scheduler/RVMThread; <init>([BLjava/lang/Thread;Ljava/lang/String;ZZI)V
        at [0x684fb914] Lorg/jikesrvm/scheduler/greenthreads/GreenThread; <init>([BLjava/lang/Thread;Ljava/lang/String;ZZI)V at line 134
        at [0x684fb914] Lorg/jikesrvm/scheduler/greenthreads/GreenThread; <init>(Ljava/lang/String;)V at line 99
        at [0x684fb914] Lorg/jikesrvm/scheduler/Scheduler$ThreadModel; <init>(Ljava/lang/String;)V at line 55
        at [0x684fb914] Lorg/jikesrvm/adaptive/controller/ControllerThread; <init>(Ljava/lang/Object;)V at line 52
        at [0x684fb914] Lorg/jikesrvm/adaptive/controller/Controller; createControllerThread()V at line 252
        at [0x684fb950] Lorg/jikesrvm/adaptive/controller/Controller; boot()V at line 177
        at [0x684fb990] Lorg/jikesrvm/VM; finishBooting()V at line 424
        at [0x684fb9c0] Lorg/jikesrvm/VM; boot()V at line 157

        Show
        dgrove David Grove added a comment - I looked at half a dozen; they all have this as the stacktrace: JikesRVM: internal error: recursive use of hardware exception registers (exiting) – Stack – at [0x684f39f8] Lorg/jikesrvm/runtime/RuntimeEntrypoints; deliverHardwareException(II)V at line 694 at [0x684f3a0c] <hardware trap> at [0x684fb8c4] Lorg/jikesrvm/scheduler/RVMThread; <init>([BLjava/lang/Thread;Ljava/lang/String;ZZI)V at [0x684fb914] Lorg/jikesrvm/scheduler/greenthreads/GreenThread; <init>([BLjava/lang/Thread;Ljava/lang/String;ZZI)V at line 134 at [0x684fb914] Lorg/jikesrvm/scheduler/greenthreads/GreenThread; <init>(Ljava/lang/String;)V at line 99 at [0x684fb914] Lorg/jikesrvm/scheduler/Scheduler$ThreadModel; <init>(Ljava/lang/String;)V at line 55 at [0x684fb914] Lorg/jikesrvm/adaptive/controller/ControllerThread; <init>(Ljava/lang/Object;)V at line 52 at [0x684fb914] Lorg/jikesrvm/adaptive/controller/Controller; createControllerThread()V at line 252 at [0x684fb950] Lorg/jikesrvm/adaptive/controller/Controller; boot()V at line 177 at [0x684fb990] Lorg/jikesrvm/VM; finishBooting()V at line 424 at [0x684fb9c0] Lorg/jikesrvm/VM; boot()V at line 157
        Hide
        dgrove David Grove added a comment -

        This seems to have gone away in the last two regression runs, but I want to leave it open until I can have a chance to force GC to happen in exactly this place again by injecting an explicit System.gc() at the right program point. I'm suspicious that there is a GC map bug here that we just weren't hitting before, we hit it for a few sanity runs, then we stopped hitting it again.

        Show
        dgrove David Grove added a comment - This seems to have gone away in the last two regression runs, but I want to leave it open until I can have a chance to force GC to happen in exactly this place again by injecting an explicit System.gc() at the right program point. I'm suspicious that there is a GC map bug here that we just weren't hitting before, we hit it for a few sanity runs, then we stopped hitting it again.
        Hide
        dgrove David Grove added a comment -

        By applying this patch to a development image, I get a 100% reproducible crash in GC during VM bootup

        Index: /home/dgrove/rvm-trunk/rvm/src/org/jikesrvm/scheduler/RVMThread.java
        ===================================================================
        — /home/dgrove/rvm-trunk/rvm/src/org/jikesrvm/scheduler/RVMThread.java (revision 14600)
        +++ /home/dgrove/rvm-trunk/rvm/src/org/jikesrvm/scheduler/RVMThread.java (working copy)
        @@ -410,6 +410,8 @@
        contextRegisters = new Registers();
        exceptionRegisters = new Registers();

        + if (VM.fullyBooted) System.gc();
        +
        if(VM.VerifyAssertions) VM._assert(stack != null);
        // put self in list of threads known to scheduler and garbage collector
        if (!VM.runningVM)

        { @@ -466,6 +468,7 @@ // create wrapper Thread if doesn't exist this.thread = java.lang.JikesRVMSupport.createThread(this, name); }

        + if (VM.fullyBooted) System.gc();
        }
        }

        Show
        dgrove David Grove added a comment - By applying this patch to a development image, I get a 100% reproducible crash in GC during VM bootup Index: /home/dgrove/rvm-trunk/rvm/src/org/jikesrvm/scheduler/RVMThread.java =================================================================== — /home/dgrove/rvm-trunk/rvm/src/org/jikesrvm/scheduler/RVMThread.java (revision 14600) +++ /home/dgrove/rvm-trunk/rvm/src/org/jikesrvm/scheduler/RVMThread.java (working copy) @@ -410,6 +410,8 @@ contextRegisters = new Registers(); exceptionRegisters = new Registers(); + if (VM.fullyBooted) System.gc(); + if(VM.VerifyAssertions) VM._assert(stack != null); // put self in list of threads known to scheduler and garbage collector if (!VM.runningVM) { @@ -466,6 +468,7 @@ // create wrapper Thread if doesn't exist this.thread = java.lang.JikesRVMSupport.createThread(this, name); } + if (VM.fullyBooted) System.gc(); } }
        Hide
        dgrove David Grove added a comment -

        Have to stop for the day. I've traced it down to what looks like fairly convincing evidence that the call to AddressArray.create in JNIEnvironment.initializeState causes the TIB of the contextRegisters object of another thread (ie, not the executing one) to get smashed, even though the objects don't appear to overlap. This happens on a prototype-opt image.even if recompilation is disabled (as long as profiling is enabled, so we create the organizer thread).

        ./dist/prototype-opt_x86_64-linux/rvm -X:aos:enable_recompilation=false -X:aos:gather_profile_data=true -X:verbose -verbose:gc

        With this instrumentation (plus some more higher in the call stack).

        protected void initializeState(RVMThread target )

        { if (VM.runningVM) VM.sysWriteln("initializeState 1 ", Magic.objectAsAddress(target.contextRegisters), " TIB ", Magic.objectAsAddress(ObjectModel.getTIB(target.contextRegisters))); if (VM.runningVM) VM.sysWriteln("this is ", Magic.objectAsAddress(this)); JNIRefs = AddressArray.create(JNIREFS_ARRAY_LENGTH + JNIREFS_FUDGE_LENGTH); VM.sysWriteln("Allocated JNIRefs array is ", Magic.objectAsAddress(JNIRefs)); VM.sysWriteln("sized ", JNIREFS_ARRAY_LENGTH + JNIREFS_FUDGE_LENGTH); VM.sysWriteln("check size ", JNIRefs.length()); if (VM.runningVM) VM.sysWriteln("initializeState 10 ", Magic.objectAsAddress(target.contextRegisters), " TIB ", Magic.objectAsAddress(ObjectModel.getTIB(target.contextRegisters))); JNIRefsTop = 0; JNIRefsSavedFP = 0; if (VM.runningVM) VM.sysWriteln("initializeState 20 ", Magic.objectAsAddress(target.contextRegisters), " TIB ", Magic.objectAsAddress(ObjectModel.getTIB(target.contextRegisters))); JNIRefsMax = (JNIREFS_ARRAY_LENGTH - 1) << LOG_BYTES_IN_ADDRESS; alwaysHasNativeFrame = false; if (VM.runningVM) VM.sysWriteln("initializeState 100 ", Magic.objectAsAddress(target.contextRegisters), " TIB ", Magic.objectAsAddress(ObjectModel.getTIB(target.contextRegisters))); }

        I see the appended output.

        allocateEnvironment 1 0x9f80014c TIB 0x600c3df4
        allocateEnvironment 10 0x9f80014c TIB 0x600c3df4
        the allocated environment is 0x70016318
        initializeState 1 0x9f80014c TIB 0x600c3df4
        this is 0x70016318
        Allocated JNIRefs array is 0x9f800018
        sized 150
        check size 150
        initializeState 10 0x9f80014c TIB 0x00000000
        initializeState 20 0x9f80014c TIB 0x00000000
        initializeState 100 0x9f80014c TIB 0x00000000
        allocateEnvironment 100 0x9f80014c TIB 0x00000000
        NEW THREAD cREG P30 0x9f80014c TIB 0x00000000

        It's 100% deterministic at least, and seems to be not impacted by adding more and more sysWrites....

        Show
        dgrove David Grove added a comment - Have to stop for the day. I've traced it down to what looks like fairly convincing evidence that the call to AddressArray.create in JNIEnvironment.initializeState causes the TIB of the contextRegisters object of another thread (ie, not the executing one) to get smashed, even though the objects don't appear to overlap. This happens on a prototype-opt image.even if recompilation is disabled (as long as profiling is enabled, so we create the organizer thread). ./dist/prototype-opt_x86_64-linux/rvm -X:aos:enable_recompilation=false -X:aos:gather_profile_data=true -X:verbose -verbose:gc With this instrumentation (plus some more higher in the call stack). protected void initializeState(RVMThread target ) { if (VM.runningVM) VM.sysWriteln("initializeState 1 ", Magic.objectAsAddress(target.contextRegisters), " TIB ", Magic.objectAsAddress(ObjectModel.getTIB(target.contextRegisters))); if (VM.runningVM) VM.sysWriteln("this is ", Magic.objectAsAddress(this)); JNIRefs = AddressArray.create(JNIREFS_ARRAY_LENGTH + JNIREFS_FUDGE_LENGTH); VM.sysWriteln("Allocated JNIRefs array is ", Magic.objectAsAddress(JNIRefs)); VM.sysWriteln("sized ", JNIREFS_ARRAY_LENGTH + JNIREFS_FUDGE_LENGTH); VM.sysWriteln("check size ", JNIRefs.length()); if (VM.runningVM) VM.sysWriteln("initializeState 10 ", Magic.objectAsAddress(target.contextRegisters), " TIB ", Magic.objectAsAddress(ObjectModel.getTIB(target.contextRegisters))); JNIRefsTop = 0; JNIRefsSavedFP = 0; if (VM.runningVM) VM.sysWriteln("initializeState 20 ", Magic.objectAsAddress(target.contextRegisters), " TIB ", Magic.objectAsAddress(ObjectModel.getTIB(target.contextRegisters))); JNIRefsMax = (JNIREFS_ARRAY_LENGTH - 1) << LOG_BYTES_IN_ADDRESS; alwaysHasNativeFrame = false; if (VM.runningVM) VM.sysWriteln("initializeState 100 ", Magic.objectAsAddress(target.contextRegisters), " TIB ", Magic.objectAsAddress(ObjectModel.getTIB(target.contextRegisters))); } I see the appended output. allocateEnvironment 1 0x9f80014c TIB 0x600c3df4 allocateEnvironment 10 0x9f80014c TIB 0x600c3df4 the allocated environment is 0x70016318 initializeState 1 0x9f80014c TIB 0x600c3df4 this is 0x70016318 Allocated JNIRefs array is 0x9f800018 sized 150 check size 150 initializeState 10 0x9f80014c TIB 0x00000000 initializeState 20 0x9f80014c TIB 0x00000000 initializeState 100 0x9f80014c TIB 0x00000000 allocateEnvironment 100 0x9f80014c TIB 0x00000000 NEW THREAD cREG P30 0x9f80014c TIB 0x00000000 It's 100% deterministic at least, and seems to be not impacted by adding more and more sysWrites....
        Hide
        zyridium Daniel Frampton added a comment -

        This error was introduced with the read barrier branch – sorry for the time it took tracking it down.

        The register fields are explicitly traced by GC through the thread table, so we must keep the new register objects alive until the thread slot has been assigned.

        Fixed in 14644.

        Show
        zyridium Daniel Frampton added a comment - This error was introduced with the read barrier branch – sorry for the time it took tracking it down. The register fields are explicitly traced by GC through the thread table, so we must keep the new register objects alive until the thread slot has been assigned. Fixed in 14644.
        Hide
        dgrove David Grove added a comment -

        reopening so I can modify fix target to 3.0

        Show
        dgrove David Grove added a comment - reopening so I can modify fix target to 3.0
        dgrove David Grove made changes -
        Field Original Value New Value
        Workflow jira [ 17583 ] X10 Workflow [ 18186 ]
        dgrove David Grove made changes -
        Priority Blocker [ 10 ] Highest [ 1 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            dgrove David Grove
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: