Trying to handle nested interrupts in IRQ mode:
(USER) You are in user mode, your PC register tells where you are.
(USER) You get an interrupt.
(IRQ) Core switches to interrupt mode, disables interrupts (mask via cpsr.i) saves PC from user mode to LR in IRQ mode.
(IRQ) At this moment you can't get more interrupts because interrupts are disabled.
(IRQ) You are in IRQ mode and you do enable interrupts.
(IRQ) You call a function to handle related interrupt. After function call LR contains, return address into top level IRQ handler.
(IRQ) Another interrupt happens, LR gets overwritten with function handling interrupt. At this moment you can't have a stable state, simply because you can't use LR for your own purposes in IRQ mode and have interrupts enabled to support nested interrupts. You think you may save LR to stack, but that would be open to a race condition which would lead to inifinitive loops.
Handling nested interrupts via System mode:
(USER) You are in user mode, your PC register tells where you are.
(USER) You get an interrupt.
(IRQ) Core switches to interrupt mode, disables interrupts (mask via cpsr.i) saves PC from user mode to LR in IRQ mode.
(IRQ) At this moment you can't get more interrupts because interrupts are disabled.
(IRQ) You switch to system mode, enable IRQs.
(SYSTEM) Another interrupt happens but things work just because you don't race with hardware to keep your LR sane.
Look below to see which registers gets banked in which states.