You need to tell Redshift to load NaN as a NULL if that is what you want to do.
For example:
COPY foo from 's3://xxx' credentials 'xxxx' DELIMETER AS ',' NULL 'NaN';
This should execute successfully and insert a NULL into the table instead of NaN.