Even after a lot of research I wasn't getting anywhere, then I accidentally landed here : http://blogs.msdn.com/b/vsod/archive/2010/04/07/how-to-implement-late-bound-event-handling-in-case-of-a-vsto-add-in.aspx
which solved my problem. For anyone else having the same problem, you basically need to do the following in assemblyinfo.cs:
1. Set ExcelLocale1033 attribute to false
2. Mark the assembly as COM Visible