Get around python passing by reference

Question

I have a small script that reads in some rows from a table and outputs some nested JSON. The issue is that since Python is pass by reference, every time I update my temp variable, the JSON object I was building gets changed. Here's what I'm taking about:

rows.py

rows = [
    ['1,', '\\WIN-2N5E8Q63188', 'Memory > Available MBytes', ''],
    ['2,', '\\WIN-2N5E8Q63188', 'Physical Disk > Avg. Disk sec/Write', '0 C:'],
    ['3,', '\\WIN-2N5E8Q63188', 'Physical Disk > Avg. Disk sec/Write', '1 F:'],
    ['4,', '\\WIN-2N5E8Q63188', 'Physical Disk > Avg. Disk sec/Read', '0 C:'],
    ['5,', '\\WIN-2N5E8Q63188', 'Physical Disk > Avg. Disk sec/Read', '1 F:'],
    ['6,', '\\WIN-2N5E8Q63188', 'Processor > % Processor Time', '_Total'],
    ['7,', '\\WIN-2N5E8Q63188', 'Processor > % Privileged Time', '_Total'],
    ['8,', '\\WIN-2N5E8Q63187', 'Memory > Available MBytes', '']
]

create_json.py

from rows import rows

machine_template = {
    'machineName': '',
    'counters': []
}

counter_template = {
    'counterName': '',
    'instances': []
}

instance_template = {
    'instanceName': '',
    'counterId': -1
}

counter_ids = []
machines = []
counters = []
instances = []

current_machine = ''
current_counter = ''
last_machine = ''
last_counter = ''
current_template = []

data = {'machines': []}

for i in rows:
    counter_ids.append(i[0])
    machines.append(i[1])
    counters.append(i[2])
    instances.append(i[3])

row_count = len(counter_ids)
i = 0

while i < row_count:
    ct = []
    mt = []
    new_machine = False
    new_counter = False

    # handle the instance
    it = instance_template
    it['instanceName'] = instances[i]
    it['counterId'] = counter_ids[i]

    # handle the counter
    current_counter = counters[i]
    if current_counter != last_counter:
        new_counter = True
        ct = counter_template
        ct['counterName'] = counters[i]

    # handle the machine name
    current_machine = machines[i]
    if current_machine != last_machine:
        new_machine = True
        mt = machine_template
        mt['machineName'] = current_machine

    # build the template
    if new_counter:
        ct['instances'].append(it)

    if new_machine:
        mt['counters'].append(ct)
        data['machines'].append(mt)

    last_counter = current_counter
    last_machine = current_machine
    i += 1

print data

Here's what the finished product should look like (not exactly for the data provided in the rows.py file, but the same structure):

{
    "machines":[
            {
                "machineName": "\\WIN-2N5E8Q63188",
                "counters": [
                    {
                        "counterName": "MemoryAvailable MBytes",
                        "instances": [
                            {
                                "instanceName": "",
                                "counterId": 1
                            }
                        ]
                    },
                    {
                        "counterName": "PhysicalDiskAvg. Disk sec/Write",
                        "instances": [
                            {
                                "instanceName": "0 C:",
                                "counterId": 2
                            },
                            {
                                "instanceName": "1 F:",
                                "counterId": 3
                            }
                        ]
                    }
                ]
            }
    ]
}

And here's what is returned from my script:

{
    'machines': [
        {
            'machineName': '\\WIN-2N5E8Q63187',
            'counters': [
                {
                    'instances': [
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        }
                    ],
                    'counterName': 'Memory>AvailableMBytes'
                },
                {
                    'instances': [
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        }
                    ],
                    'counterName': 'Memory>AvailableMBytes'
                }
            ]
        },
        {
            'machineName': '\\WIN-2N5E8Q63187',
            'counters': [
                {
                    'instances': [
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        }
                    ],
                    'counterName': 'Memory>AvailableMBytes'
                },
                {
                    'instances': [
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        },
                        {
                            'counterId': '8,
                            ',
                            'instanceName': ''
                        }
                    ],
                    'counterName': 'Memory>AvailableMBytes'
                }
            ]
        }
    ]
}

Any ideas of how I can make this work?

Answer 1

You'll need to make copies of the templates and possibly other lists that you intend to reuse, however as your templates themselves contain empty lists, a shallow copy (dict.copy()) is not enough. Use deepcopy from the copy module:

from copy import deepcopy

... later ...

# handle the instance
it = deepcopy(instance_template)
it['instanceName'] = instances[i]
it['counterId'] = counter_ids[i]

# handle the counter
current_counter = counters[i]
if current_counter != last_counter:
    new_counter = True
    ct = deepcopy(counter_template)
    ct['counterName'] = counters[i]

# handle the machine name
current_machine = machines[i]
if current_machine != last_machine:
    new_machine = True
    mt = deepcopy(machine_template)
    mt['machineName'] = current_machine

This will leave the original template untouched and safe for reuse.

You might also find the pprint module helpful for printing dicts:

from pprint import pprint
pprint(data)

Result:

{'machines': [{'counters': [{'counterName': 'Memory > Available MBytes',
                             'instances': [{'counterId': '1,',
                                            'instanceName': ''}]}],
               'machineName': '\\WIN-2N5E8Q63188'},
              {'counters': [{'counterName': 'Memory > Available MBytes',
                             'instances': [{'counterId': '8,',
                                            'instanceName': ''}]}],
               'machineName': '\\WIN-2N5E8Q63187'}]}

Like I said, you might also need to copy any lists that get reused - it's a little hard to tell if this output is correct without better understanding of your code.