Here's what worked:
rsync -uvma --include="*/" --include="templates/**.html" --exclude="*" $html_all_dir $html_dir
My guess is, your format and mine probably accomplish the same thing. I know I tried about 20 different patterns before this one, and this is the only one that worked properly. I don't think I tried your format though :)