Regex Replacing : to “:” etc

Question

I've got a bunch of strings like:

"Hello, here's a test colon:. Here's a test semi-colon&#59;"

I would like to replace that with

"Hello, here's a test colon:. Here's a test semi-colon;"

And so on for all printable ASCII values.

At present I'm using boost::regex_search to match &#(\d+);, building up a string as I process each match in turn (including appending the substring containing no matches since the last match I found).

Can anyone think of a better way of doing it? I'm open to non-regex methods, but regex seemed a reasonably sensible approach in this case.

Thanks,

Dom

Answer 1

The big advantage of using a regex is to deal with the tricky cases like & Entity replacement isn't iterative, it's a single step. The regex is also going to be fairly efficient: the two lead characters are fixed, so it will quickly skip anything not starting with &#. Finally, the regex solution is one without a lot of surprises for future maintainers.

I'd say a regex was the right choice.

Is it the best regex, though? You know you need two digits and if you have 3 digits, the first one will be a 1. Printable ASCII is after all  -~. For that reason, you could consider &#1?\d\d;.

As for replacing the content, I'd use the basic algorithm described for boost::regex::replace :

For each match // Using regex_iterator<>
    Print the prefix of the match
    Remove the first 2 and last character of the match (&#;)
    lexical_cast the result to int, then truncate to char and append.

Print the suffix of the last match.

Answer 2

This will probably earn me some down votes, seeing as this is not a c++, boost or regex response, but here's a SNOBOL solution. This one works for ASCII. Am working on something for Unicode.

        NUMS = '1234567890'
MAIN    LINE = INPUT                                :F(END)
SWAP    LINE ?  '&#' SPAN(NUMS) . N ';' = CHAR( N ) :S(SWAP)
        OUTPUT = LINE                               :(MAIN)
END

Answer 3

* Repaired SNOBOL4 Solution
* & -> &
     digit = '0123456789'
main line = input                        :f(end)
     result = 
swap line arb . l
+    '&#' span(digit) . n ';' rem . line :f(out)
     result = result l char(n)           :(swap)
out  output = result line                :(main)
end

Answer 4

The existing SNOBOL solutions don't handle the multiple-patterns case properly, due to there only being one "&". The following solution ought to work better:

        dd = "0123456789"
        ccp = "#" span(dd) $ n ";" *?(s = s char(n)) fence (*ccp | null)
   rdl  line = input                              :f(done)
   repl line "&" *?(s = ) ccp = s                 :s(repl)
        output = line                             :(rdl)
   done
   end

Answer 5

I don't know about the regex support in boost, but check if it has a replace() method that supports callbacks or lambdas or some such. That's the usual way to do this with regexes in other languages I'd say.

Here's a Python implementation:

s = "Hello, here's a test colon:. Here's a test semi-colon&#59;"
re.sub(r'&#(1?\d\d);', lambda match: chr(int(match.group(1))), s)

Producing:

"Hello, here's a test colon:. Here's a test semi-colon;"

I've looked some at boost now and I see it has a regex_replace function. But C++ really confuses me so I can't figure out if you could use a callback for the replace part. But the string matched by the (\d\d) group should be available in $1 if I read the boost docs correctly. I'd check it out if I were using boost.

Answer 6

Ya know, as long as we're off topic here, perl substitution has an 'e' option. As in evaluate expression. E.g.

echo "Hello, here's a test colon:. Here's a test semi-colon&#59;
Further test A. abc.~.def."
| perl -we 'sub translate { my $x=$_[0]; if ( ($x >= 32) && ($x <= 126) )
{ return sprintf("%c",$x); } else { return "&#".$x.";"; } }
while (<>) { s/&#(1?\d\d);/&translate($1)/ge; print; }'

Pretty-printing that:

#!/usr/bin/perl -w

sub translate
{
  my $x=$_[0];

  if ( ($x >= 32) && ($x <= 126) )
  {
    return sprintf( "%c", $x );
  }
  else
  {
    return "&#" . $x . ";" ;
  }
}

while (<>)
{
  s/&#(1?\d\d);/&translate($1)/ge;
  print;
}

Though perl being perl, I'm sure there's a much better way to write that...

---

Back to C code:

You could also roll your own finite state machine. But that gets messy and troublesome to maintain later on.

Answer 7

Here's another Perl's one-liner (see @mrree's answer):

• a test file:

$ cat ent.txt 
Hello,  here's a test colon:. 
Here's a test semi-colon&#59; 'ƒ'

• the one-liner:

$ perl -pe's~&#(1?\d\d);~
> sub{ return chr($1) if (31 < $1 && $1 < 127); $& }->()~eg' ent.txt

• or using more specific regex:

$ perl -pe"s~&#(1(?:[01][0-9]|2[0-6])|3[2-9]|[4-9][0-9]);~chr($1)~eg" ent.txt

• both one-liners produce the same output:

Hello,  here's a test colon:. 
Here's a test semi-colon; 'ƒ'

Answer 8

boost::spirit parser generator framework allows easily to create a parser that transforms desirable NCRs.

// spirit_ncr2a.cpp
#include <iostream>
#include <string>
#include <boost/spirit/include/classic_core.hpp>

int main() {
  using namespace BOOST_SPIRIT_CLASSIC_NS; 

  std::string line;
  while (std::getline(std::cin, line)) {
    assert(parse(line.begin(), line.end(),
         // match "&#(\d+);" where 32 <= $1 <= 126 or any char
         *(("&#" >> limit_d(32u, 126u)[uint_p][&putchar] >> ';')
           | anychar_p[&putchar])).full); 
    putchar('\n');
  }
}

• compile:

    $ g++ -I/path/to/boost -o spirit_ncr2a spirit_ncr2a.cpp

• run:

    $ echo "Hello,  here's a test colon:." | spirit_ncr2a

• output:

    "Hello,  here's a test colon:." 

Answer 9

I did think I was pretty good at regex but I have never seen lambdas been used in regex, please enlighten me!

I'm currently using python and would have solved it with this oneliner:

''.join([x.isdigit() and chr(int(x)) or x for x in re.split('&#(\d+);',THESTRING)])

Does that make any sense?

Answer 10

Here's a NCR scanner created using Flex:

/** ncr2a.y: Replace all NCRs by corresponding printable ASCII characters. */
%%
&#(1([01][0-9]|2[0-6])|3[2-9]|[4-9][0-9]); { /* accept 32..126 */
  /**recursive: unput(atoi(yytext + 2)); skip '&#'; `atoi()` ignores ';' */
  fputc(atoi(yytext + 2), yyout); /* non-recursive version */
}

To make an executable:

$ flex ncr2a.y
$ gcc -o ncr2a lex.yy.c -lfl

Example:

$ echo "Hello,  here's a test colon:. 
> Here's a test semi-colon&#59; 'ƒ'
> &#38;#59; <-- may be recursive" \
> | ncr2a

It prints for non-recursive version:

Hello,  here's a test colon:.
Here's a test semi-colon; 'ƒ'
&#59; <-- may be recursive

And the recursive one produces:

Hello,  here's a test colon:.
Here's a test semi-colon; 'ƒ'
; <-- may be recursive

Answer 11

This is one of those cases where the original problem statement apparently isn't very complete, it seems, but if you really want to only trigger on cases which produce characters between 32 and 126, that's a trivial change to the solution I posted earlier. Note that my solution also handles the multiple-patterns case (although this first version wouldn't handle cases where some of the adjacent patterns are in-range and others are not).

      dd = "0123456789"
      ccp = "#" span(dd) $ n *lt(n,127) *ge(n,32) ";" *?(s = s char(n))
 +      fence (*ccp | null)
 rdl  line = input                              :f(done)
 repl line "&" *?(s = ) ccp = s                 :s(repl)
      output = line                             :(rdl)
 done
 end

It would not be particularly difficult to handle that case (e.g. ;#131;#58; produces ";#131;:" as well:

      dd = "0123456789"
      ccp = "#" (span(dd) $ n ";") $ enc
 +      *?(s = s (lt(n,127) ge(n,32) char(n), char(10) enc))
 +      fence (*ccp | null)
 rdl  line = input                              :f(done)
 repl line "&" *?(s = ) ccp = s                 :s(repl)
      output = replace(line,char(10),"#")       :(rdl)
 done
 end

Answer 12

Here's a version based on boost::regex_token_iterator. The program replaces decimal NCRs read from stdin by corresponding ASCII characters and prints them to stdout.

#include <cassert>
#include <iostream>
#include <string>
#include <boost/lexical_cast.hpp>
#include <boost/regex.hpp>

int main()
{
  boost::regex re("&#(1(?:[01][0-9]|2[0-6])|3[2-9]|[4-9][0-9]);"); // 32..126
  const int subs[] = {-1, 1}; // non-match & subexpr
  boost::sregex_token_iterator end;
  std::string line;

  while (std::getline(std::cin, line)) {
    boost::sregex_token_iterator tok(line.begin(), line.end(), re, subs);

    for (bool isncr = false; tok != end; ++tok, isncr = !isncr) {
      if (isncr) { // convert NCR e.g., '&#58;' -> ':'
        const int d = boost::lexical_cast<int>(*tok);
        assert(32 <= d && d < 127);
        std::cout << static_cast<char>(d);
      }
      else
        std::cout << *tok; // output as is
    }
    std::cout << '\n';
  }
}