You have to apply the function to all of them induvidually. So
fit4 = lm(mut ~ log(ent) + log(score) + log(wt) +
log(I(ent^2)) + log(I(score^2)) +log(I(wt^2)))
will do the desired
Reason:
log(ent + score + wt + I(ent^2) + I(score^2) +I(wt^2))
is interpreted as a single regressor.
So to r it is like lm(mut~x)
where x=log(...)
instead of
x=log(ent) + ... + log(I(wt^2))