I ended up with just preprocessing input outside of Prolog and then using read/1
.
Also, info from B-Prolog creator Neng-Fa Zhou (https://groups.google.com/forum/?fromgroups#!topic/comp.lang.prolog/KSXc2QTWZck):
You can use an internal predicate, called read_next_token(Type,Val), for your purpose. Another predicate, called b_NEXT_TOKEN_ff(Type,Val), does the same thing but is faster. These predicates are not documented.