In order to achieve your goal, you have to change some entries in the Registry. Preview HAndlers have GUIDs that are stored in the Registry. For each filetype, if there is one available, there is a preview handler.
For .csv files you need the Preview Handler from Microsoft Excel since a .csv file is Excel.