WinMerge: How to compare files with the same content but different encodings?

Question

Motivation: I am rewriting a doc -- text files to be processed later. The new sources now use UTF-8. Large portions of the sources are the same. I need to find differences.

Details: The old doc sources use the cp1250 encoding, the new sources use the UTF-8. Both new and old sources use the same line endings (CR+LF). I am using the Unicode version of the WinMerge application (WinMergeU.exe), version 2.12.4.0.

It almost works, but... When the lines differ, they are initially marked as block by the dark yellow, and the different portions are marked using the lighter colour. When moving the red block cursor there, the panes below show the different part.

However, the block of text is marked by the dark yellow also in cases when (the Unicode representation of) the text is the same. The red block moves also to those portions of the files. In such case, the two panes below (that show the differences) containt the same text and nothing is marked as different. See the picture below:

The very first line differs -- this is OK. But the second line has visually the same content. The only character outside of the ASCII range is Ú there. It has a different representation in the encoded sources. This causes the line marked as different, but the panes below does not mark anyting at the line as different.

See also the following paragraphs that are exactly the same (only the encoding in the sources differ, the same line ending is used).

It looks as if the initial comparison were based on binary representation of the lines. Is there any setting to tell WinMerge that the comparison (I mean the block marking) should be based on Unicode content?

I tried hard, but no luck, yet.

Update: The above question was for the latest stable 2.12.4. The beta version 2.13.22 works just perfectly for me. See my answer below.

Answer 1

I think it really should not be the task of a merge tool to allow the merging of files stored in different encodings.

An encoding is a function that maps bytes (stored on the disk or in memory) to characters (displayed on screen). Unfortunately, by default the encoding of a file is not stored together with the file. Therefore, any program that wants to open the file and display its contents needs to guess the encoding. While this sometimes works, it is also an error prone procedure.

Now, the character sets of different encodings do not overlap in general. So what is the merge tool supposed to do if you merge a character C from file A in encoding X into a file B in encoding Y, if character C is not part of the character set of encoding Y?

Thus, I think the task of a merge tool should be to merge the binary content. Anything else is a dirty hack and damned to fail at some level. (A merge tool maker may decide to provide character level merging, which also might work most of the time. But there is some guesswork involved.)

Therefore, I'd also recommend you first translate the old files to UTF-8 and then merge those with the new versions.

Answer 2

This doesn't really answer your question about WinMerge, but have you considered using another diff program? One of my favorites is kdiff - http://kdiff3.sourceforge.net/

When I do a compare on KDiff using one UTF8 file and another Unicode file, I get the following:

Here is the compare screen - note that the encodings on the files are different, but the files are considered to be equal from a text standpoint:

Answer 3

Just for your information. The question was for the latest stable 2.12.4. I have tried the beta version 2.13.22, and it works just perfectly for me. See the difference for exactly the same files -- only the first lines in the files were removed. (My big thanks to authors.)

Answer 4

1. Edit -> Options
2. Select 'Compare' from categories pane on left.
3. Check box 'Ignore carriage return differences' (UNIX, Windows, Mac)

Answer 5

I would recommend converting the files to the same encoding before diffing.

If you are working with a version control system I'd recommend the following:

1. Create a fresh checkout of the files
2. Convert all files to UTF-8
3. Commit the files
4. Copy your new files over
5. Use WinMerge

That way you end up with two commits in the history - one for the encoding change and another for the content changes and WinMerge will work as expected.

Answer 6

What about option File -> File Encoding... in WinMerge? It allows to set encoding for files independently.