So I guess these includes resolve to something like /usr/include etc. Is there any way to fix this without changing all the includes from to "foo.h" manually?
Yes, sure.
First, you would need to write CONFIG += serialport
as per documentation.
You could also specify the includepath to the qtserialport clone on your system.
You would still need to get the library built, but I assume that you are aware of it.