We don't have x87 exceptions enabled by default, this means that operations that use the x87 stack may unexpectedly get NaN values from the stack if it has overflowed. The overflows may occur from bugs elsewhere in the RVM or in native code.
An example of a problem would be, imagine there are values on the x87 stack, we take an exception, during the exception we fail to reset the x87 stack state, after delivering the exception we put something on the x87 stack say for a conversion, as the stack is full the value of NaN is pushed, NaN propagates throughout the code causing unexpected behavior.
It has been observed that some benchmarks with lots of floating point code intermittently fail, this may be the cause. We should enable the bit in the x87 control world that generates exceptions for SSE code only, as in x87 code the exception could be from a divide by 0 which Java treats as creating a NaN.