How to query Alert System Enable mail profile properties

Question 1

Future-proofing is a very wise idea :). As you noted, both xp_regread and xp_instance_regread are also undocumented. And http://social.msdn.microsoft.com/Forums/sqlserver/en-US/b83dd2c1-afde-4342-835f-c1debd73d9ba/xpregread explains your concern (plus, it offers you an alternative).

Your trace and your run of sp_helptext 'sp_get_sqlagent_properties' are a good start. The next thing to do is run sp_helptext 'sp_helptext', and note its reference to sys.syscomments. BOL sys.syscomments topic redirects you to sys.sql_modules, and that points to the next step. Unfortunately for your needs, just one row (for 'sp_get_sqlagent_properties') will be returned by running USE msdb; SELECT object_name(object_id) FROM sys.sql_modules WHERE definition LIKE '%sp_get_sqlagent_properties%'. I thus assume you are out of luck - there appears to be no alternative, publicly documented, module (sproc). My assumption could be wrong :).

I deduce that xp_reg% calls exist for client (SMO, SSMS, etc.) needs, such as setting/getting agent properties. More importantly (for your needs), your sp_helptext run also reveals SSMS (a client) is using a registry store (i.e. not a SQL store). Unfortunately, I must deduce (based upon an absence of proof from a library search) that those keys (and their values) are also not documented...

The above appears to put you in a pickle. You could decide "if we are going to rely upon undocumented registry keys, we might as well rely on the undocumented calls to read them", but I won't recommend that:). You could also file a feature request at https://connect.microsoft.com/ (your need is clear), but because your need concerns a client-side feature request, I do not recommend holding your breath while waiting for a fix :).

Perhaps it is time to step back and take a look at the bigger picture:

How often can that key be changed, and how often will this process poll for that change?
Email uses a mail primitive. Sender: "Dear recipient, did you get my mail?" Recipient: "Dear sender, did you send me mail?" Disabling an email profile is not the only reason for an email failure.

Would a different approach be more useful, when compared to periodically checking a key?

One approach would be to periodically send "KeepAlive" email. If the "KeepAlive" email isn't periodically received, maybe that key was tweaked, maybe the Post Office is on a holiday, or maybe something else (equally bad) happened. Plus, this approach should be fully supported, documented, and be future-proof. Who knows how and what keys will be used in the next version of SQL Server Agent?

The first bullet isn't addressed (neither would it be addressed by periodically checking a key), and perhaps you have additional needs (worth mentioning on MS connect).

Question 2

I finally found a way to do this using PowerShell and Microsoft.SqlServer.Management.Smo.Agent.JobServer.

Here is the PowerShell script I wrote to check if SQL Agent mail alerts are enabled, and to make sure that SQL Agent is set to Auto startup when the server reboots. This works works with local or remote SQL instances.

# usage examples
Check-SQLAgentConfiguration -InstanceName 'localhost\sql2014'
Check-SQLAgentConfiguration -InstanceName 'RemoteServerName'


function Check-SQLAgentConfiguration{
    param([string]$InstanceName='localhost')

    [System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.SMO') | out-null;

    $smosrv = new-object Microsoft.SqlServer.Management.Smo.Server($InstanceName);
    $smosrv.ConnectionContext.ConnectTimeout = 5; #5 seconds timeout.
    $smosrv.ConnectionContext.ApplicationName = 'PowerShell Check-SQLAgentConfiguration';
    "Server: {0}, Instance: {1}, Version: {2}, Product Level: {3}" -f $smosrv.Name, $smosrv.InstanceName, $smosrv.Version, $smosrv.ProductLevel;

    # NOTE: this does not seem to ever fail even if SQL Server is offline.
    if(!$smosrv){"SQL Server Connection failed";  return $null;}   

    $agent = $smosrv.JobServer;

    if(!$agent -or $agent -eq $null){
        throw "Agent Connection failed";  
        return -2;
    }

    $agentConfigErrMsg = "";

    if($agent.AgentMailType -ne "DatabaseMail"){ $agentConfigErrMsg += " AgentMailType: " + $agent.AgentMailType + "; "; }
    if(!$agent.DatabaseMailProfile){$agentConfigErrMsg += " DatabaseMailProfile: " + $agent.DatabaseMailProfile + "; ";}
    if($agent.SqlAgentAutoStart -ne "True"){$agentConfigErrMsg += " SqlAgentAutoStart: " + $agent.SqlAgentAutoStart + " ServiceStartMode: " + $agent.ServiceStartMode + "; "; }

    if($agentConfigErrMsg.length -gt 0){
       $agentConfigErrMsg = "Invalid SQL Agent config! " + $agentConfigErrMsg; 
       throw $agentConfigErrMsg;
       return -1;
    }

    <##
    #for debugging:
    "Valid: "
    "AgentMailType:" + $agent.AgentMailType;
    "DatabaseMailProfile: " + $agent.DatabaseMailProfile;
    "ServiceStartMode: " + $agent.ServiceStartMode;
    "SqlAgentAutoStart: " + $agent.SqlAgentAutoStart;
    #"SqlAgentMailProfile: " + $agent.SqlAgentMailProfile;
    #>

    return 0; 
}

Question 3

SQL 2008R2 uses Service-broker like queues for mail processing (http://technet.microsoft.com/en-us/library/ms175887%28v=sql.105%29.aspx). In our environments I check that the corresponding queue exists and is active.

SELECT * FROM msdb.sys.service_queues
WHERE name = N'ExternalMailQueue' 
AND is_receive_enabled = 1;

This table is listed online (http://technet.microsoft.com/en-us/library/ms187795%28v=sql.105%29.aspx).

Testing shows that this makes the required transition as we went from new instance -> enabled mail -> mail switched off and back again.