mongodb query data where the key is variable and aggregate results using python

Question

i'm currently working with a saltstack implementation that is using a mongodb returner and i'm trying to query the results that are being stored in it for reporting purposes. unfortunately the way that the returner is writing information to the database looks like the following:

db.hostname21.find({"20140421035007474763" : {$exists : 1}}, {_id: 0}).pretty()

{
     "fun" : "state.sls",
     "20140421035007474763" : {
            "file_|-Get-Logins-Stats_|-/scripts/server_logins_stats_|-managed" : {
                    "comment" : "File /scripts/server_logins_stats is in the correct state",
                    "__run_num__" : 2,
                    "changes" : {

                    },
                    "name" : "/scripts/server_logins_stats",
                    "result" : true
            },
            "service_|-Logger_|-rsyslog_|-running" : {
                    "comment" : "Service rsyslog is already enabled, and is in the desired state",
                    "__run_num__" : 1,
                    "changes" : {

                    },
                    "name" : "rsyslog",
                    "result" : true
            },
            "cmd_|-Run_Script_|-/scripts/server_logins_stats_|-run" : {
                    "comment" : "Command \"/scripts/server_logins_stats\" run",
                    "__run_num__" : 4,
                    "changes" : {
                            "pid" : 20899,
                            "retcode" : 0,
                            "stderr" : "",
                            "stdout" : "0"
                    },
                    "name" : "/scripts/server_logins_stats",
                    "result" : true
            },
            "cron_|-Schedule_Run_|-/scripts/server_logins_stats_|-present" : {
                    "comment" : "Cron /scripts/server_logins_stats already present",
                    "__run_num__" : 3,
                    "changes" : {

                    },
                    "name" : "/scripts/server_logins_stats",
                    "result" : true
            },
            "pkg_|-nc_|-nc_|-installed" : {
                    "comment" : "Package nc is already installed",
                    "__run_num__" : 0,
                    "changes" : {

                    },
                    "name" : "nc",
                    "result" : true
            }
    }
}

as you can see here the keys are constantly changing instead of having a key that identifies each script. however, i have found that the failures are formatted pretty consistently, except that they don't have any keys identifying that it is a failure just an array of strings:

       "20140421041507478163" : [
            "Pillar failed to render with the following messages:",
            "Specified SLS 'globals' in environment 'Production' is not available on the salt master"
       ],
       "fun" : "state.sls"

so ultimately what i want to do is to be able to report on each failure by identifying the job, the host and the nature of the failure as well as the aggregate number of successes and failures over time. if you notice each host (minion) has its own collection created. so i've got a python script that will iterate over the collections to determine if the job was actually executed on that host:

import datetime
import pymongo


#hosts = ["mongoDBStaging", "mongoDBUAT"]
hosts = ["mongodbuat"]

for host in hosts:
    conn = pymongo.Connection(host)
    mdb = conn['salt']
    collections = set(mdb.collection_names())


    hosts = []
    jids = []

    # for every collection, identify whether it is a host or a job
    for c in collections:   
        # if the collection is a host add it to the host array
        if not (len(c) == 20 and int(c)):
            #print "{0} is a host".format(c)
            hosts.append(c)
        # other  wise add it to the job array
        else:
            #print "{0} is a jid".format(c)
            jids.append(c)


    for h in hosts:
        # for every job in a host connect to that collection 
        # and search for the job id to see if it exists 
        # and what its return was so we can report on that info
        for j in jids:
            coll = conn['salt'][h]
            #print "%s collection, %s jid" % (coll, j)
            for doc in coll.find({j: {'$exists': True}, "fun": "state.sls"}, {"_id": 0}):
                print "{0}".format(coll)
                print "{0} is a doc".format(doc)

but i'm struggling to be able to query into the results to actually determine whether it was successful or not. by being able to pull out elements of the document that is returned to read into the results from each document.

if anyone has any suggestions on how i could consistently query into results where the keys are constantly changing to get to the results that would be really helpful.

Answer 1

In case anyone is wondering. I solved my own problem by using the following python code. Definitely not the best for performance or getting the most out of mongodb but it does work. I'd probably recommend updating the salt returner to use mongodb better given that querying is pretty limited within the shell.

import datetime
import pymongo
import json
import re

hosts = ["mongodbuat"]

# initialize failures and successes
failures = 0
successes = 0

for host in hosts:
    conn = pymongo.Connection(host)
    mdb = conn['salt']
    collections = set(mdb.collection_names())

    hosts = []
    jids = []

    # for every collection, identify whether it is a host or a job
    for c in collections:   
        # if the collection is a host add it to the host array
        if not (len(c) == 20 and int(c)):
            hosts.append(c)
        # otherwise add it to the job array
        else:
            jids.append(c)


    for h in hosts:
        # for every job in a host connect to that collection 
        # and search for the job id to see if it exists 
        # and what its return was so we can report on that info
        for j in jids:
            coll = conn['salt'][h]

            # search for the json documents returned from mongodb
            # if the jobid exists in that host
            for doc in coll.find({j: {'$exists': True}, "fun": "state.sls"}, {"_id": 0}):

                # if the jobid exists find the host name in a readable format
                c = str(coll)
                thishost = ''
                match = re.search('(\w+)\.spottrading\.com',c)
                if match:
                    thishost = match.group(1)

                # search the document returned in the form of a dictionary for
                # the states you want to report on
                for jobid, states in doc.iteritems():
                    if re.search('\d+', jobid):
                        print '\njob id =', jobid
                        if isinstance(states, list):
                            print states
                            failures += 1
                        elif isinstance(states, dict):
                            for job, data in states.iteritems():
                                print '\tjob: {0}, result: {1}'.format(job, data[u'result'])
                            successes += 1

print "{0} successes, {1} failures".format(successes, failures)