You could touch
an intermediate file and use it to carry the information:
ssh user@hostname "touch file -r remote-file"
Then scp -p
the new file back:
scp -p user@hostname:file local_destination
Then touch -r
the file you copied in the first place.