I can list a number of projects that can help you in this process:
d3js, which you already mentioned. It is not only limited to visualizing but also looking at the API section of the library, offers many manipulation methods.
DataSet offers a good way to manipulate data.
TableTop is a library intended for parsing of CSV data
jStat is a library for statistical data analysis.
With a bit of googling I also found this interesting website, full of infos: https://jgoodall.me/2012/02/01/javascript-statistical-libraries