Your problem is that you're using the "-" operator on strings, not with lead/lag. I think you want this:
select seqno,
set_a,
lag_val,
lead_val
, case
when (lag_val ) is null then '('
when (lead_val ) is null then ')'
when (lag_val = set_a) then 'and'
else 'or'
end as menu_entry
from ( select seqno,
set_a,
lag(set_a,1) over (order by seqno) as lag_val,
-- lag(to_number,'XX') over (order by seqno) as lag_val,
lead(set_a,1) over (order by seqno) as lead_val
from menu_items)
order by seqno