You can look into MEF, so your end users would provide DLL modules in a certain format (see the link I mentioned), which would later be discovered in your program and executed at any given time.
Or use a math parser:
- VB.Net- Evaluating Mathematical Expression in a String
- Evaluate mathematical expression from a string using VB
But I feel that approach #1 would be more flexible.