Is it possible to use the Hadoop SDK, especially LINQ to Hive, with a local installation of HDInsight Server. Note that I am not refering to HDInsight Service hosted on Azure.
I tried to use LINQ to Hive from Microsoft.Hadoop.Hive Nuget package, but was unable to get it working, because LINQ to Hive seems to require that results are stored in Azure Blob Storage, rather than on my hosted instance.
var hiveConnection = new HiveConnection(new Uri("http://hadoop-poc.cloudapp.net:50111"), "hadoop", "hgfhdfgh", "hadoop", "hadooppartner", "StorageKey");
var metaData = hiveConnection.GetMetaData().Result;
var result = hiveConnection.ExecuteQuery(@"select * from customer limit 1");
Even with a storage key, I cannot get this to work, because the MapReduce job fails with:
AzureException: org.apache.hadoop.fs.azure.AzureException: Container a7e3aa39-75ba-4cc2-a8aa-301257018146 in account hadooppartner not found, and we can't create it using anoynomous credentials.
I also added the credentials once more to the core-site.xml
file, as follows:
<property>
<name>fs.azure.account.key.hadooppartner.blob.core.windows.net</name>
<value>Credentials</value>
</property>
However I would rather get rid of storing results on Azure Storage, if possible.
Thank you for your help!