Hive on HDInsight with certificate issues

Question

I was going through http://azure.microsoft.com/en-us/documentation/articles/hdinsight-use-hive/ but get

Start-AzureHDInsightJob : Request failed after (12) attempts over a period of (00:03:10.4800236) with code: Forbidden
Content:<Error xmlns="http://schemas.microsoft.com/windowsazure" 
xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Code>ForbiddenError</Code><Message>The server failed to authenticate 
the request. Verify that the certificate is valid and is associated with this subscription.</Message></Error>
At D:\sandbox\hadoop\hive\ProcessExampleSampleLog.ps1:15 char:12
+ $hiveJob = Start-AzureHDInsightJob -Cluster $clusterName -JobDefinition $hiveJob ...
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Start-AzureHDInsightJob], HttpLayerException
    + FullyQualifiedErrorId : Microsoft.WindowsAzure.Management.HDInsight.Framework.Core.Library.WebRequest.HttpLayerExce 
   ption,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.StartAzureHDInsightJobCmdlet

I checked the certificate via Get-AzureSubscription -Current which gave me:

SubscriptionName          : Dev
SubscriptionId            : <CORRECT SUBSCRIPTION ID>
<SNIP>
IsDefault                 : False
Certificate               : [Subject]
                              CN=Kevin

                            [Issuer]
                              CN=Kevin

                            [Serial Number]
                              <REDACTED>

                            [Not Before]
                              13/05/2013 14:27:18

                            [Not After]
                              19/04/2113 14:27:18

                            [Thumbprint]
                              <THUMBPRINT THAT MATCHES IN AZURE PORTAL WITH DEV SUBSCRIPTION>

CurrentStorageAccountName : <DIFFERENT STORAGE NAME TO THE HDINSIGHT ONE>
ActiveDirectoryUserId     : <MY EMAIL>
TokenProvider             : Microsoft.WindowsAzure.Commands.Utilities.Common.Authentication.AdalTokenProvider

The only thing that doesn't match is that the CurrentStorageAccountName above isn't the one I'm using for HDInsight but that's set in the query.

I executed Add-AzureAccount successfully prior to all this also.

We changed the name of our subscriptions recently (but before I created the HDInsight cluster and its storage account) which may be causing this.

I did delete my certificate via the azure portal and reuploaded it.

I'm now a bit stuck.

Answer 1

According to this http://blogs.msdn.com/b/bigdatasupport/archive/2013/11/21/getting-started-with-hdinsight-powershell-tools-and-sdk.aspx I hadn't imported the azure publish settings file.

---

Getting Azure Management certificate via Publishsettings file:

On each workstation you plan to use the HDInsight SDK (PowerShell or .Net SDK), you can use the following steps to obtain an Azure management certificate-

1. Sign in to the Windows Azure Management Portal using the credentials for your Windows Azure account.

2. Once the logon to Azure is complete and Portal is open, run the Windows Azure PowerShell command to get the settings file –

   Get-AzurePublishSettingsFile

   The Get-AzurePublishSettingsFile cmdlet opens a web page on the [Windows Azure Management Portal] from which you can download the subscription information. The information is contained in a .publishsettings file.

3. Import the Azure settings file to be used by Windows Azure cmdlets, by running the cmdlet –

   Import-AzurePublishSettingsFile '<Folder>\YourSubscriptionName-DownlodDate-credentials.publishsettings'

   Here, '<Folder>\YourSubscriptionName-DownlodDate-credentials.publishsettings' is the file you saved in step 2 on your workstation.

Import-AzurePublishSettingsFile cmdlet does two things -

1. It parses this AzurePublishSettingsFile XML file, reads the certificate data and installs that certificate in your local certificate store (usually Current User/Personal) -The certificate has 'Windows Azure Tools' as 'Issued to' and 'Issued By'.

2. It create a file called 'WindowsAzureProfile.xml' under the folder 'C:\Users\userName\AppData\Roaming\Windows Azure PowerShell' - The file contains Subscription Name, SubscriptionId and Azure certificate Thumbprint etc.

---

So now you know.