I have no idea what plugin.url_for() does, but you should be able to speed it a bit doit it this way:
plugin_url_for = plugin.url_for
_, dirs, _ = iter(os.walk(PATH_TO_DIRECTORY)).next()
items = [{
'label': directory, 'path': plugin_url_for('test')
} for directory in dirs]
I dropped root, files variables as it seems you are not using it, also removed enumerate on dirs as you are not making any use of it. However, put it back if you need it for some weird reason. Please test it and let me know if it helped. I can not test it properly myself for obvious reasons.