Why are some unintended symbols added to my string?

Question

I wrote a console application which fetches strings from some fields in a Sharepoint list. Then I simply write the strings to console. This works fine for the most fields. There is one MultiLineTextField with RichText enabled where i had to remove all the html-tags, that causes this issue.

Even after all the tags are removed the strings seem to contain question marks which were never added to the string. The most weird thing about this is when I set a breakpoint and look into the string's value there are no question marks, but they suddenly appear on the console output.

The only thing I could think of was to Trim the string. Because sometimes they appear in front of the actual string sometimes they are at the and of it, but never in between.

So this is what I tried:

myString = myString.Trim();
myString = myString.Replace("?",string.Empty);

But this does not solve the issue. Besides this would not be a smart solution in case one of the strings would be supposed to contain question marks. For detailed code please see the link above.

Also Convert.ToBase64String(Encoding.UTF8.GetBytes(myString)) gives me the following output:

4oCLTWVobCwgRWllciwgV2Fzc2VyLCBIYWNrZmxlaXNjaCA=

Answer 1

There are probably some non-printing unicode (or possibly low ASCII) characters in the end of the string. The console has a different encoding, and will often render such as ?. Basically: use the indexer (yourString[n]) or yourString.ToCharArray() to investigate what is actually in the string aroung the location of the ?.

---

With the edit, we can see that the string has a zero-width space (decimal 8203) at the start:

Answer 2

Sounds like you're maybe having a problem with unicode characters. Chances are you're outputting the string as ASCII instead of Unicode. Take a look at this question as it sounds like you may be experiencing the same problem.