Your second code is correct.
This Formats my string like this "HISTOIRE D\'O, CHAPITRE II";
No, it really doesn't. That's how it looks in the debugger, but if you dump it to a console or something similar, you'll see there's only a single backslash.
As an alternative, you could use a verbatim string literal:
test = test.Replace("'", @"\'");