You can use
select sqrt(val2*val1)/val1 from t where val1 != 0
;
That will avoid the rows where val1 is 0 and so will not cause the "divide by 0" error. This should give you the second format.
For the first format where you want to show the 0 in the output you can use a case statement in the where clause.
select id,
case
when val1 = 0 then 0
when val1 != 0 then sqrt(val2*val1)/val1
end
from t