This is entirely normal and not a problem. Reference assemblies are only used by the compiler to retrieve metadata, the declarations of the types in the assembly. The equivalent of .h files in native C++. Those declarations do not depend on the target architecture.
Architecture dependency is resolved at runtime. For .NET Framework assemblies that contain native code (mscorlib.dll, System.Data.dll, PresentationFramework.dll), the correct assembly is retrieved from a different GAC subdirectory. The GAC_32 subdirectory contains assemblies with 32-bit native code, GAC_64 for 64-bit, GAC_MSIL has the assemblies that contain pure IL and therefore have no architecture dependency.