How do you fix a TeamCity shared resource lock after a hard crash?

Question

Background: I have a TeamCity installation with dozens of build configurations. Many of these build configurations deploy to the same servers. To alleviate any conflicts, I used TeamCity's shared resources feature to control how these builds run. All the builds that deploy to one particular server require a write lock for the same shared resource; thus, they won't try to deploy to the same server at the same time.

Over the weekend we had a hard crash due to a power outage. The crash happened while a build, which had a write lock on a shared resource, was running. Now it appears that write lock never got released. None of the builds that use that shared resource can run--they all just queue up. If I disable the resource and then re-enable it, a build will run, but the next time it will just queue up again.

I would really rather not delete and re-create these shared resources; each one is used by 6-9 build configurations, and re-assigning them all would be a huge headache. How to I clear the un-released write lock?

Answer 1

Nick, shared resource lock state is determined in runtime and is not stored anywhere. So, if there is a hanging build that requires a write lock, this can cause described behaviour. In this case you should check for hanging builds and forcibly terminate them for lock to be released

Also, there is an issue TW-36042 in TeamCity 8.1.2 that causes resources with infinite quota to behave wrongly with write locks. If your behaviour matches that in the issue - the workaround (before 8.1.3 update) is to convert infinite resource to resource with specified quota