You can certainly run an HDInsight box locally. This is separate from the Azure storage and computer emulation, and is installed through the Web Platform installer (just search for HDInsight).
There are some subtle differences between the local and Azure version, in that the local version works with data stored in HDFS, whereas in the cloud you can use Azure Blob Containers. As far as developing and testing you transform processes (in MapReduce / Hive / Pig) this make no real difference. The only difference is the way you would get the data in and out.
Note that you can certainly create MapReduce jobs with C# on HDInsight, for basic data transformations it can be a lot easier to use a higher level language like Pig, or possibly the SQL based HiveQL on HDInsight.