The answer was pretty simple. In versions before 2.2, the play framework created the startup script at location target/start
so your Heroku Procfile would have looked like
web: target/start -Dhttp.port=$PORT $JAVA_OPTS
This changed in Play 2.2 and the script is now at location target/universal/stage/bin/foo
so your Procfile should look like this
web: target/universal/stage/bin/foo -Dhttp.port=$PORT $JAVA_OPTS
This has also been explained in this SO answer
In my case, the old application was started because the build output was never cleaned, so I successfully started the old version after building the new one instead of getting a target/start
file not found error.
The heroku-repo plugin let me clear the build cache, which lead to the target/start
file not found error, which let to the solution.