After countless hours, it was of course something stupid.
When you load the XACT3 .XGS, .XSB, .XWB files into the solution explorer(Content in my case), you MUST also make sure to click on the properties of each one and make sure that Build Action is set to NONE.
The error was thrown because XNA didn't know how to build those files, and in true MS fashion, the errors are usually quite confusing and never really take you to the real problem.
Also, for future visitors, make sure you also set Copy to Output to ALWAYS.