You have the solution wrong. You can't have code that in one hand returns a string, on the other returns a number. Thats never sensible! Think in the case where you wanted to use this field in further expressions later on etc - not just the case of displaying the field on the report.
The solution is simply to return NA() rather than TEXT("N/A") Which will return null. Then set the ifnull flag on the numeric field to be "N/A". That works. Because ifnull is just the display property of the field this works fine.