From TechNote.txt in the WinRAR installation folder:
- To process an SFX archive you need to skip the SFX module searching for the marker block in the archive. There is no marker block sequence (0x52 0x61 0x72 0x21 0x1a 0x07 0x00) in the SFX module itself.
This you already indicated yourself: the unrar source code has in archive.cpp the code how it reads in a file. The maximum size of the SFX part will never be larger than the number defined in rardefs.hpp:
#define MAXSFXSIZE 0x100000