Is there a UNIX/ELF equivalent?
No. UNIX has no concept of a GUI or CUI Subsystem, so the question "what subsystem does this executable use?" is completely meaningless.
Question
In Windows you can check if IMAGE_OPTIONAL_HEADERS.Subsystem from the PE header is equal to either IMAGE_SUBSYSTEM_WIN_GUI or IMAGE_SUBSYSTEM_WIN_CUI.
Is there a UNIX/ELF equivalent?
I've checked
https://en.wikipedia.org/wiki/Executable_and_Linkable_Format#File_header
along with general Google and
http://linux.die.net/man/5/elf
a few times but I haven't found anything.
Solution
Is there a UNIX/ELF equivalent?
No. UNIX has no concept of a GUI or CUI Subsystem, so the question "what subsystem does this executable use?" is completely meaningless.