EDIT: I think the problem might be the command you're using to set the parameter. Try:
Write-Host "##teamcity[setParameter name='TestParameter' value='2']"
--
We've experienced the same behavior. The key here is 'subsequent steps.' You must modify the parameter in a separate build step that is run before the step in which you want to use the new parameter.
It's my understanding that all parameters in a build step are evaluated immediately before the execution of that step. The tokens will be replaced with the unmodified values of those parameters. Thus, what actually gets executed by the build agent is:
Write-Host "TestParameter value is 1"
Write-Host "##teamcity[setParameter name='TestParameter' value='2']"
Write-Host "TestParameter value is 1"