Apparently there was another line matching the favicon, hidden in an include file, which I ommitted in the question for the sake of simplicity.
location = /favicon.ico {
log_not_found off;
access_log off;
}
Removing this (because there is favicon present on this site) solved my issue :-)