Undesired character encoding translation in transform output

Question

I have a situation here.

-JAVA version 1.6.
-Linux RH 5.6 Tikanga.
-Windows 7 (Cygwin latest version)
-Gradle 1.6

On one machine (Linux) = when I run Gradle build, I get resultant .xsl files which are similar to what I was getting when I was running ANT build

On another Linux machine = I see differences in the .xsl files.

What all System level files in Linux and even in Windows and any environment variables can affect this kind of behavior?

In every case, I get a successful build but resultant .xsl files after transformation are different on 2 different Linux machines and in Windows (using Cygwin).

$LANG ? -- is set to en_US.UTF-0 in all machines (Linux/Windows)
$LC_ALL ? -- this is blank 

$  cat /etc/mime.types |grep xsl
application/xml                 xml xsl

On one of the Linux machine, I see this value as

text/xml        xml xsl

Expand.xsl which is used during the transformation looks like:

<?xml version="1.0" encoding="UTF-8"?><xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0" id="automaticinterceptservice">

<xsl:output method="xml" encoding="UTF-8"/>

<xsl:template match="xsl:include">
    <xsl:apply-templates select="document(@href)/*"/>
</xsl:template>

<xsl:template match="@*|node()">
    <xsl:copy>
    <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
</xsl:template>

</xsl:stylesheet>

In the following image snapshot, left side file is what ANT build process generated and right side is what I'm getting from Gradle. If I run the same source code Gradle build on another Linux machine, I don't see any difference in the .xsl files after the build.

Command locale output: Machine 1 - Linux - where .xsl file transforms without any differences.

-bash-3.2$ locale -a
aa_DJ
aa_DJ.iso88591
aa_DJ.utf8
aa_ER
aa_ER@saaho
aa_ER.utf8
aa_ER.utf8@saaho
aa_ET
aa_ET.utf8
af_ZA
af_ZA.iso88591
af_ZA.utf8
am_ET
am_ET.utf8
an_ES
an_ES.iso885915
an_ES.utf8
ar_AE
ar_AE.iso88596
ar_AE.utf8
ar_BH
ar_BH.iso88596
ar_BH.utf8
ar_DZ
ar_DZ.iso88596
ar_DZ.utf8
ar_EG
ar_EG.iso88596
ar_EG.utf8
ar_IN
ar_IN.utf8
ar_IQ
ar_IQ.iso88596
ar_IQ.utf8
ar_JO
ar_JO.iso88596
ar_JO.utf8
ar_KW
ar_KW.iso88596
ar_KW.utf8
ar_LB
ar_LB.iso88596
ar_LB.utf8
ar_LY
ar_LY.iso88596
ar_LY.utf8
ar_MA
ar_MA.iso88596
ar_MA.utf8
ar_OM
ar_OM.iso88596
ar_OM.utf8
ar_QA
ar_QA.iso88596
ar_QA.utf8
ar_SA
ar_SA.iso88596
ar_SA.utf8
ar_SD
ar_SD.iso88596
ar_SD.utf8
ar_SY
ar_SY.iso88596
ar_SY.utf8
ar_TN
ar_TN.iso88596
ar_TN.utf8
ar_YE
ar_YE.iso88596
ar_YE.utf8
as_IN.utf8
az_AZ.utf8
be_BY
be_BY.cp1251
be_BY@latin
be_BY.utf8
be_BY.utf8@latin
bg_BG
bg_BG.cp1251
bg_BG.utf8
bn_BD
bn_BD.utf8
bn_IN
bn_IN.utf8
bokmal
bokmål
br_FR
br_FR@euro
br_FR.iso88591
br_FR.iso885915@euro
br_FR.utf8
bs_BA
bs_BA.iso88592
bs_BA.utf8
byn_ER
byn_ER.utf8
C
ca_AD
ca_AD.iso885915
ca_AD.utf8
ca_ES
ca_ES@euro
ca_ES.iso88591
ca_ES.iso885915@euro
ca_ES.utf8
ca_FR
ca_FR.iso885915
ca_FR.utf8
ca_IT
ca_IT.iso885915
ca_IT.utf8
catalan
croatian
csb_PL
csb_PL.utf8
cs_CZ
cs_CZ.iso88592
cs_CZ.utf8
cy_GB
cy_GB.iso885914
cy_GB.utf8
czech
da_DK
da_DK.iso88591
da_DK.iso885915
da_DK.utf8
danish
dansk
de_AT
de_AT@euro
de_AT.iso88591
de_AT.iso885915@euro
de_AT.utf8
de_BE
de_BE@euro
de_BE.iso88591
de_BE.iso885915@euro
de_BE.utf8
de_CH
de_CH.iso88591
de_CH.utf8
de_DE
de_DE@euro
de_DE.iso88591
de_DE.iso885915@euro
de_DE.utf8
de_LU
de_LU@euro
de_LU.iso88591
de_LU.iso885915@euro
de_LU.utf8
deutsch
dutch
dz_BT
dz_BT.utf8
eesti
el_CY
el_CY.iso88597
el_CY.utf8
el_GR
el_GR.iso88597
el_GR.utf8
en_AU
en_AU.iso88591
en_AU.utf8
en_BW
en_BW.iso88591
en_BW.utf8
en_CA
en_CA.iso88591
en_CA.utf8
en_DK
en_DK.iso88591
en_DK.utf8
en_GB
en_GB.iso88591
en_GB.iso885915
en_GB.utf8
en_HK
en_HK.iso88591
en_HK.utf8
en_IE
en_IE@euro
en_IE.iso88591
en_IE.iso885915@euro
en_IE.utf8
en_IN
en_IN.utf8
en_NZ
en_NZ.iso88591
en_NZ.utf8
en_PH
en_PH.iso88591
en_PH.utf8
en_SG
en_SG.iso88591
en_SG.utf8
en_US
en_US.iso88591
en_US.iso885915
en_US.utf8
en_ZA
en_ZA.iso88591
en_ZA.utf8
en_ZW
en_ZW.iso88591
en_ZW.utf8
es_AR
es_AR.iso88591
es_AR.utf8
es_BO
es_BO.iso88591
es_BO.utf8
es_CL
es_CL.iso88591
es_CL.utf8
es_CO
es_CO.iso88591
es_CO.utf8
es_CR
es_CR.iso88591
es_CR.utf8
es_DO
es_DO.iso88591
es_DO.utf8
es_EC
es_EC.iso88591
es_EC.utf8
es_ES
es_ES@euro
es_ES.iso88591
es_ES.iso885915@euro
es_ES.utf8
es_GT
es_GT.iso88591
es_GT.utf8
es_HN
es_HN.iso88591
es_HN.utf8
es_MX
es_MX.iso88591
es_MX.utf8
es_NI
es_NI.iso88591
es_NI.utf8
es_PA
es_PA.iso88591
es_PA.utf8
es_PE
es_PE.iso88591
es_PE.utf8
es_PR
es_PR.iso88591
es_PR.utf8
es_PY
es_PY.iso88591
es_PY.utf8
es_SV
es_SV.iso88591
es_SV.utf8
estonian
es_US
es_US.iso88591
es_US.utf8
es_UY
es_UY.iso88591
es_UY.utf8
es_VE
es_VE.iso88591
es_VE.utf8
et_EE
et_EE.iso88591
et_EE.iso885915
et_EE.utf8
eu_ES
eu_ES@euro
eu_ES.iso88591
eu_ES.iso885915@euro
eu_ES.utf8
fa_IR
fa_IR.utf8
fi_FI
fi_FI@euro
fi_FI.iso88591
fi_FI.iso885915@euro
fi_FI.utf8
finnish
fo_FO
fo_FO.iso88591
fo_FO.utf8
français
fr_BE
fr_BE@euro
fr_BE.iso88591
fr_BE.iso885915@euro
fr_BE.utf8
fr_CA
fr_CA.iso88591
fr_CA.utf8
fr_CH
fr_CH.iso88591
fr_CH.utf8
french
fr_FR
fr_FR@euro
fr_FR.iso88591
fr_FR.iso885915@euro
fr_FR.utf8
fr_LU
fr_LU@euro
fr_LU.iso88591
fr_LU.iso885915@euro
fr_LU.utf8
fy_NL
fy_NL.utf8
ga_IE
ga_IE@euro
ga_IE.iso88591
ga_IE.iso885915@euro
ga_IE.utf8
galego
galician
gd_GB
gd_GB.iso885915
gd_GB.utf8
german
gez_ER
gez_ER@abegede
gez_ER.utf8
gez_ER.utf8@abegede
gez_ET
gez_ET@abegede
gez_ET.utf8
gez_ET.utf8@abegede
gl_ES
gl_ES@euro
gl_ES.iso88591
gl_ES.iso885915@euro
gl_ES.utf8
greek
gu_IN
gu_IN.utf8
gv_GB
gv_GB.iso88591
gv_GB.utf8
hebrew
he_IL
he_IL.iso88598
he_IL.utf8
hi_IN
hi_IN.utf8
hr_HR
hr_HR.iso88592
hr_HR.utf8
hrvatski
hsb_DE
hsb_DE.iso88592
hsb_DE.utf8
hu_HU
hu_HU.iso88592
hu_HU.utf8
hungarian
hy_AM
hy_AM.armscii8
hy_AM.utf8
icelandic
id_ID
id_ID.iso88591
id_ID.utf8
is_IS
is_IS.iso88591
is_IS.utf8
italian
it_CH
it_CH.iso88591
it_CH.utf8
it_IT
it_IT@euro
it_IT.iso88591
it_IT.iso885915@euro
it_IT.utf8
iw_IL
iw_IL.iso88598
iw_IL.utf8
ja_JP
ja_JP.eucjp
ja_JP.ujis
ja_JP.utf8
japanese
japanese.euc
ka_GE
ka_GE.georgianps
ka_GE.utf8
kk_KZ
kk_KZ.pt154
kk_KZ.utf8
kl_GL
kl_GL.iso88591
kl_GL.utf8
km_KH
km_KH.utf8
kn_IN
kn_IN.utf8
ko_KR
ko_KR.euckr
ko_KR.utf8
korean
korean.euc
ku_TR
ku_TR.iso88599
ku_TR.utf8
kw_GB
kw_GB.iso88591
kw_GB.utf8
ky_KG
ky_KG.utf8
lg_UG
lg_UG.iso885910
lg_UG.utf8
lithuanian
lo_LA
lo_LA.utf8
lt_LT
lt_LT.iso885913
lt_LT.utf8
lv_LV
lv_LV.iso885913
lv_LV.utf8
mai_IN
mai_IN.utf8
mg_MG
mg_MG.iso885915
mg_MG.utf8
mi_NZ
mi_NZ.iso885913
mi_NZ.utf8
mk_MK
mk_MK.iso88595
mk_MK.utf8
ml_IN
ml_IN.utf8
mn_MN
mn_MN.utf8
mr_IN
mr_IN.utf8
ms_MY
ms_MY.iso88591
ms_MY.utf8
mt_MT
mt_MT.iso88593
mt_MT.utf8
nb_NO
nb_NO.iso88591
nb_NO.utf8
ne_NP
ne_NP.utf8
nl_BE
nl_BE@euro
nl_BE.iso88591
nl_BE.iso885915@euro
nl_BE.utf8
nl_NL
nl_NL@euro
nl_NL.iso88591
nl_NL.iso885915@euro
nl_NL.utf8
nn_NO
nn_NO.iso88591
nn_NO.utf8
no_NO
no_NO.iso88591
no_NO.utf8
norwegian
nr_ZA
nr_ZA.utf8
nso_ZA
nso_ZA.utf8
nynorsk
oc_FR
oc_FR.iso88591
oc_FR.utf8
om_ET
om_ET.utf8
om_KE
om_KE.iso88591
om_KE.utf8
or_IN
or_IN.utf8
pa_IN
pa_IN.utf8
pa_PK
pa_PK.utf8
pl_PL
pl_PL.iso88592
pl_PL.utf8
polish
portuguese
POSIX
pt_BR
pt_BR.iso88591
pt_BR.utf8
pt_PT
pt_PT@euro
pt_PT.iso88591
pt_PT.iso885915@euro
pt_PT.utf8
romanian
ro_RO
ro_RO.iso88592
ro_RO.utf8
ru_RU
ru_RU.iso88595
ru_RU.koi8r
ru_RU.utf8
russian
ru_UA
ru_UA.koi8u
ru_UA.utf8
rw_RW
rw_RW.utf8
se_NO
se_NO.utf8
sid_ET
sid_ET.utf8
si_LK
si_LK.utf8
sk_SK
sk_SK.iso88592
sk_SK.utf8
slovak
slovene
slovenian
sl_SI
sl_SI.iso88592
sl_SI.utf8
so_DJ
so_DJ.iso88591
so_DJ.utf8
so_ET
so_ET.utf8
so_KE
so_KE.iso88591
so_KE.utf8
so_SO
so_SO.iso88591
so_SO.utf8
spanish
sq_AL
sq_AL.iso88591
sq_AL.utf8
sr_CS
sr_CS.iso88595
sr_CS.utf8
sr_ME
sr_ME.utf8
sr_RS
sr_RS@latin
sr_RS.utf8
sr_RS.utf8@latin
ss_ZA
ss_ZA.utf8
st_ZA
st_ZA.iso88591
st_ZA.utf8
sv_FI
sv_FI@euro
sv_FI.iso88591
sv_FI.iso885915@euro
sv_FI.utf8
sv_SE
sv_SE.iso88591
sv_SE.iso885915
sv_SE.utf8
swedish
ta_IN
ta_IN.utf8
te_IN
te_IN.utf8
tg_TJ
tg_TJ.koi8t
tg_TJ.utf8
thai
th_TH
th_TH.tis620
th_TH.utf8
ti_ER
ti_ER.utf8
ti_ET
ti_ET.utf8
tig_ER
tig_ER.utf8
tl_PH
tl_PH.iso88591
tl_PH.utf8
tn_ZA
tn_ZA.utf8
tr_CY
tr_CY.iso88599
tr_CY.utf8
tr_TR
tr_TR.iso88599
tr_TR.utf8
ts_ZA
ts_ZA.utf8
tt_RU.utf8
turkish
uk_UA
uk_UA.koi8u
uk_UA.utf8
ur_PK
ur_PK.utf8
uz_UZ
uz_UZ@cyrillic
uz_UZ.iso88591
uz_UZ.utf8@cyrillic
ve_ZA
ve_ZA.utf8
vi_VN
vi_VN.tcvn
vi_VN.utf8
wa_BE
wa_BE@euro
wa_BE.iso88591
wa_BE.iso885915@euro
wa_BE.utf8
xh_ZA
xh_ZA.iso88591
xh_ZA.utf8
yi_US
yi_US.cp1255
yi_US.utf8
zh_CN
zh_CN.gb18030
zh_CN.gb2312
zh_CN.gbk
zh_CN.utf8
zh_HK
zh_HK.big5hkscs
zh_HK.utf8
zh_SG
zh_SG.gb2312
zh_SG.gbk
zh_SG.utf8
zh_TW
zh_TW.big5
zh_TW.euctw
zh_TW.utf8
zu_ZA
zu_ZA.iso88591
zu_ZA.utf8
-bash-3.2$

OK, while performing the transformations manually.. one of the coworker found this:

I have been experimenting with the character encoding of the gradle XSL files and I think I know how the encoding is getting mixed up. The tests below are using the ant/gradle outputs from my cygwin environment. I am using notepad++ to perform the encoding changes.

How to “fix” one of the files output by the gradle build:
1. The file should initially show as encoded with UTF-8 without BOM.
2. Convert the file to ANSI (Encoding -> Convert to ANSI)
3. Set the encoding to UTF-8 without BOM (Encoding -> Encode in UTF-8 without BOM)
4. All the odd characters are gone, and the file is bit-identical to the original ant output. (verified with MD5 + SHA-1)

How to “break” one of the files output by the ant build:
1. The file should initially show as encoded with UTF-8 without BOM.
2. Set the encoding to ANSI. (Encoding -> Encode in ANSI)
3. Convert the file to UTF-8 without BOM. (Encoding -> Convert to UTF-8 without BOM)
4. The odd characters are present, and the file is bit-identical to the original gradle output. (verified with MD5 + SHA-1)

Answer 1

OK.. it seems like I don't need jvmArgs line in the above xsltCleanup task, IF the following 2 variables are SET in ~/.bashrc. Having jvmArgs line in the task solved DEV local builds but it still didn't resolve the behavior when Jenkins build (CM Team) had LANG set to something different than UTF-8 (at Jenkins/System).

export LANG=en_US.UTF-8
export GRADLE_OPTS="-Dfile.encoding=UTF-8"

Making the above change in my ~/.bashrc and ~/.bash_profile (calling ~/.bashrc).. fully resolved the issue for local builds in Windows(Cygwin) and in a local Linux. Setting the above two variable/properties in Jenkins GLOBAL Settings config page, did the trick for Jenkins build as well. One can also set these at the job config level.

Arun

Answer 2

OK, we found the Shenzie.

In Jenkins somehow file.encoding SYSTEM Property was changed from "UTF-8" default for that machine to

file.encoding   ANSI_X3.4-1968

On the boxes where Gradle build was generating correct .xsl files (without any differences from ANT generated .xsl files)... this was set correctly to UTF-8

Also, the following SYSTEM property in JENKINS was set to ANSI.. as well

sun.jnu.encoding    ANSI_X3.4-1968

which in my local Jenkins/Linux machine where things were good, was set to UTF-8.

Corporate policy usually is to use UTF-8 instead of ANSI (if I'm correct).

.... the root cause and permanent fix solution to this whole problem was during XSLCleanup task.. if we pass the JVM argument to force use -Dfile.encoding=UTF-8, then we never have to depend upon system's default file.encoding. In the following code snapshot (Gradle), we never used jvmArgs "-Dfile.encoding=UTF-8" line in the Gradle code.. even though "args" to javaexec was passing 'UTF-8' to it. Anyways... putting jvmArgs line fixed the issue, no more funky characters now.

// Cleanup style sheets in work directory
def performXsltCleanup() {
   println '- performXsltCleanup'
   javaexec {
      classpath = configurations.thidsXslt
      jvmArgs "-Dfile.encoding=UTF-8"
      main = "com.thc.ids.presentation.tools.devtools.preprocess.XslClean"
      args = [ "$buildDir/tmpXsl/xsl-expanded", 'UTF-8', 'quiet' ]
   }
}