I experienced and found a deadlock issue in the optimizing compiler when ObjectConstantOperand.similar() is called and a concurrent thread is waiting to compile a method transitively invoked by similar().
Reminder: to compile a method M, the current thread needs to acquire M's lock, and then acquire the compiler's lock.
ObjectConstantOperand.similar(Operand) is a method invoked while the optimizing compiler is active.
o1.similar(o2) will call o1.value.equals(o2.value), where value.equals() is potentially an application method.
Thus when similar() is called by the optimizing compiler, the current thread T1 may reenter the application. In the meantime, a concurrent thread T2 may want to compile a method M, so it acquires the lock on M, however since the compiler's lock is held by T1, it has to block.
Now, let say T1 reaches a call to M which is still not compiled. T1 tries to acquire M's lock, but it is held by T2, so T1 has to block. T1 and T2 are in a deadlock situation.
I frequently observe the deadlock while running Dacapo 9.12 Xalan.