Your assumption is correct.
We can prove it by contradiction:
Suppose the LRU candidate (the byte with most zero bits) is c and has a bit set in position x.
This means that line x has not been used after line c, therefore x must have all the zero bits that c has, plus a zero in position x. This is a contradiction as c is the byte with most zero bits so we conclude that c cannot have any bit set.