As you mentioned, the time required to invoke the command remotely must also take into account the time needed to deserialize any given object you return from your remote pipeline (in your case the whole tree of FileSystemInfo of the C drive). I would suggest to limit the number of objects you serialize and deserialize over the network and, while comparing performances of your servers, perhaps consider the time taken from within the machine you are running the code on:
Invoke-Command -comp LOCALSERVER { Measure-Command { gci -recurse "C:\" } }