TL;DR: Generally, it's best to eliminate unnecessarily visible headers when developing C++ or C.
Is there a better way of doing it?
It's generally a very bad idea when your project is not small. Most sources won't need to see the declaration of most files, and those are high level objects. At least divide it by category.
Does this slow down compile time much?
Generally, yes. Example: Do most of your source files need to know about the UI library? Probably not.
Is this a better solution?
Yes. The forward declarations' cost is very very small.
Also, I am aware that there is something called precompiled headers, but have never used myself. Is this perhaps a solution to my problem and something I should look into?
For some builds, they can save a lot of time. For others, they may only slow things down. If you choose to use them, then measure the build times to confirm which is better for your project and the includes you need.
Typically, you find all your sources don't need to know about something very high level and large (e.g. a GUI library and abstraction layer). Only a portion of the sources generally need to know about those higher level libraries. This is often a good point to break up your codebase into smaller targets/libraries.
...and to summarize all questions. What would you have done in my place?
I wouldn't stuff all those high level libraries/headers in a header for all sources to see. It's painful to undo in large codebases and tends to cost a lot in build times along the way. Headers containing forward declarations are good.