Microsoft .NET Compiler Platform, knows as Roslyn was open sourced couple weeks ago.
Go to http://roslyn.codeplex.com and get source code for both C# and VB.NET compilers. You can browser the source online at http://source.rosly.codeplex.com as well.
When you have the code, you can generate diagrams, find out how it finds tokens, etc.