Teamcity NuGet Repository corruption

Question

So, my google-fu is weak... I could not find another instance of my errors.

I've been having problems with my teamcity nuget repository since a day. NuGet downloads from the repo fail with unexpected EOF's or corrupt package warnings.

As far as I can tell this is not a hardware failure, the vm, and vm-host do not report disk errors.

To add insult to injury the teamcity logfile 'teamcity-javaLogging-2013-07-17.log' grows unbounded (+3GB in the time to type this, where 10 MB a day is normal) with the stacktraces like the ones below.

My Teamcity version is 7.1.5 (build 24400)

Anyone know how to recover from this failure?

I've not yet summoned up enough courage to just clear all the caches I can find on the teamcity admin page ( Administration > Diagnostics > Caches) because there is a warning in scary yellow on that page not to do that.

Below is a sample of the stacktraces I'm getting.

17-jul-2013 3:00:02 net.sf.ehcache.store.DiskStore get
SEVERE: provider-nugetCache: Could not read disk store element for key 2731. Error was unexpected EOF in middle of data block
java.io.StreamCorruptedException: unexpected EOF in middle of data block
        at java.io.ObjectInputStream$BlockDataInputStream.refill(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.read(Unknown Source)
        at java.io.DataInputStream.readInt(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown Source)
        at java.io.ObjectInputStream.readInt(Unknown Source)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.SerializableEntry.readSplitted(SerializableEntry.java:5)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.EntryImpl.readObjectInternal(EntryImpl.java:34)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.SerializableEntry.readExternal(SerializableEntry.java:16)
        at java.io.ObjectInputStream.readExternalData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.readObject(Unknown Source)
        at net.sf.ehcache.store.DiskStore.loadElementFromDiskElement(DiskStore.java:313)
        at net.sf.ehcache.store.DiskStore.get(DiskStore.java:268)
        at net.sf.ehcache.Cache.searchInDiskStore(Cache.java:1290)
        at net.sf.ehcache.Cache.get(Cache.java:904)
        at net.sf.ehcache.Cache.get(Cache.java:879)
        at jetbrains.buildServer.serverSide.metadata.impl.cache.TypedCacheImpl.getValue(TypedCacheImpl.java:3)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.MetadataStorageImpl.getReportedKeys(MetadataStorageImpl.java:7)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.MetadataStorageImpl.removeBuild(MetadataStorageImpl.java:45)
        at jetbrains.buildServer.serverSide.metadata.impl.indexer.BuildIndexCleaner.performCleanup(BuildIndexCleaner.java:16)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.cleanupExtensionsData(HistoryEntryCleaner.java:38)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.performCleanup(HistoryEntryCleaner.java:138)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.performCleanup(HistoryEntryCleaner.java:132)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$3.performCleanup(ServerCleanupManagerImpl.java)
        at jetbrains.buildServer.serverSide.db.DBFacade$1$1.doInConnection(DBFacade.java:178)
        at jetbrains.buildServer.serverSide.db.DBFacade$6.doInConnection(DBFacade.java:415)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:341)
        at jetbrains.buildServer.serverSide.db.DBFacade._runSql(DBFacade.java:411)
        at jetbrains.buildServer.serverSide.db.DBFacade.access$000(DBFacade.java:33)
        at jetbrains.buildServer.serverSide.db.DBFacade$1.doInTransaction(DBFacade.java:174)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
        at jetbrains.buildServer.serverSide.db.DBFacade.compact(DBFacade.java:171)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.startCleanup(ServerCleanupManagerImpl.java:74)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$2.run(ServerCleanupManagerImpl.java:0)
        at java.util.TimerThread.mainLoop(Unknown Source)
        at java.util.TimerThread.run(Unknown Source)
17-jul-2013 3:00:02 net.sf.ehcache.store.DiskStore remove
SEVERE: provider-nugetCache: Could not remove disk store entry for key 2731. Error was unexpected EOF in middle of data block
java.io.StreamCorruptedException: unexpected EOF in middle of data block
        at java.io.ObjectInputStream$BlockDataInputStream.refill(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.read(Unknown Source)
        at java.io.DataInputStream.readInt(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown Source)
        at java.io.ObjectInputStream.readInt(Unknown Source)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.SerializableEntry.readSplitted(SerializableEntry.java:5)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.EntryImpl.readObjectInternal(EntryImpl.java:34)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.SerializableEntry.readExternal(SerializableEntry.java:16)
        at java.io.ObjectInputStream.readExternalData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.readObject(Unknown Source)
        at net.sf.ehcache.store.DiskStore.loadElementFromDiskElement(DiskStore.java:313)
        at net.sf.ehcache.store.DiskStore.remove(DiskStore.java:483)
        at net.sf.ehcache.Cache.remove(Cache.java:1465)
        at net.sf.ehcache.Cache.remove(Cache.java:1392)
        at net.sf.ehcache.Cache.remove(Cache.java:1350)
        at net.sf.ehcache.Cache.remove(Cache.java:1328)
        at jetbrains.buildServer.serverSide.metadata.impl.cache.TypedCacheImpl.remove(TypedCacheImpl.java:16)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.MetadataStorageImpl.removeBuild(MetadataStorageImpl.java:30)
        at jetbrains.buildServer.serverSide.metadata.impl.indexer.BuildIndexCleaner.performCleanup(BuildIndexCleaner.java:16)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.cleanupExtensionsData(HistoryEntryCleaner.java:38)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.performCleanup(HistoryEntryCleaner.java:138)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.performCleanup(HistoryEntryCleaner.java:132)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$3.performCleanup(ServerCleanupManagerImpl.java)
        at jetbrains.buildServer.serverSide.db.DBFacade$1$1.doInConnection(DBFacade.java:178)
        at jetbrains.buildServer.serverSide.db.DBFacade$6.doInConnection(DBFacade.java:415)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:341)
        at jetbrains.buildServer.serverSide.db.DBFacade._runSql(DBFacade.java:411)
        at jetbrains.buildServer.serverSide.db.DBFacade.access$000(DBFacade.java:33)
        at jetbrains.buildServer.serverSide.db.DBFacade$1.doInTransaction(DBFacade.java:174)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
        at jetbrains.buildServer.serverSide.db.DBFacade.compact(DBFacade.java:171)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.startCleanup(ServerCleanupManagerImpl.java:74)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$2.run(ServerCleanupManagerImpl.java:0)
        at java.util.TimerThread.mainLoop(Unknown Source)
        at java.util.TimerThread.run(Unknown Source)

Answer 1

As far as I know, TeamCity NuGet Server is subject to the artifact cleanup policy defined in "Administration | Project-related Settings | Build History Clean-up", so make sure your package is still there!!

My personal advise is ALWAYS setup a dedicated nuget server. You can setup one for free simply cloning the official NugetGallery project on GitHub: that's the same codebase used by nuget.org so you'll have a familiar UI and increased performances (NugetGallery leverage Lucene.NET indexing capabilities).

Answer 2

As Remco said, you can clear the package cache, by going to http://{teamcity}/admin/admin.html?item=diagnostics&tab=cache and clicking "reset" next to "buildsMetadata". This will remove all the NuGet packages from your feed until you reindex.

You can reindex the NuGet package(s) generated by an individual build by calling TeamCity's REST API. To reindex all builds, you'd have to write a script to loop over all the builds and reindex each.

Bug reports:
http://youtrack.jetbrains.com/issue/TW-25384
http://youtrack.jetbrains.com/issue/TW-23576

Sample reindex script:
http://youtrack.jetbrains.com/issue/TW-19411#comment=27-408230

Answer 3

It seems that the index for the TeamCity NuGet Server got corrupted somehow, the packages were fine, so clearing the cache was the solution.

However, I didn't figure out how the index could be repopulated.

Be warned, clearing the nuget package cache removes all packages from the TeamCity NuGet Server so you start with a clean slate...