When you are trying to write WINDOW_STYLE_PATH to your file, then WINDOW_FILE_SEPARATOR is recognized as escaping sequence character. So you can change the path in unix style before writing it to the file. Pathconvert will assist you to convert the path...
<pathconvert property="new" dirsep="/">
<path location="${basedir}"/>
</pathconvert>
<echo message="${new}"/>
<replaceregexp replace="custom.dir=${new}/tasks/custom" byline="true" file="${basedir}/configuration.properties">
<regexp pattern="custom.dir=(.*)"/>
</replaceregexp>