PSConfig.exe throws NullReferenceException when trying to upgrade/patch SP2013

Question

I'm trying to complete SharePoint 2013 upgrade by using PSConfig after Windows Update has downloaded the updates, but can't get the process to complete without errors and this is causing me a major headache.

Health Analyzer shows "Product / patch installation or server upgrade required" error that says I should use either PSConfigUI.exe or this command to perform an upgrade:

PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures

but this always gets stuck at 10% of configuration task 5 (or, in later retries, simply fails despite getting to 100% - see below).

Here's what I tried and nothing works:

• Assigned permissions in regedit as described here - still stuck at 10% / fails with an error.
• Running Configuration Wizard UI (PSConfigUI.exe) - won't complete (step 9 fails).
• Clearing the config cache and running Psconfig.exe -cmd upgrade -inplace b2b -wait -force as described here - won't work.

This is the output of PSConfig.exe:

SharePoint Products Configuration Wizard version 15.0.4569.1503. Copyright (C) Microsoft Corporation 2012. All rights reserved.

Performing configuration task 1 of 4 Initializing SharePoint Products upgrade...

Waiting to get a lock to upgrade the farm.

Successfully initialized the SharePoint Products configuration.

Performing configuration task 2 of 4 Initiating the upgrade sequence...

Successfully initiated the upgrade sequence.

Performing configuration task 3 of 4 Upgrading SharePoint Products...

100.00% : Failed to upgrade SharePoint Products.

An exception of type System.NullReferenceException was thrown. Additional exception information: Object reference not set to an instance of an object.

Total number of configuration settings run: 3 Total number of successful configuration settings: 2 Total number of unsuccessful configuration settings: 1 Successfully stopped the configuration of SharePoint Products. Configuration of SharePoint Products failed.

This appears a couple of times in the log:

   Task upgrade has failed with an unknown exception 
Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.SharePoint.Upgrade.SPContentDatabaseIntegrity.CheckAppOrphans()
   at Microsoft.SharePoint.Upgrade.SPContentDatabaseIntegrity.Check(SPContentDatabaseIntegrityCheckOptions options)
   at Microsoft.SharePoint.Upgrade.SPContentDatabaseSequence.ContentDatabaseDataIntegrityCheck()
   at Microsoft.SharePoint.Upgrade.SPContentDatabaseSequence.PreUpgrade()
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively)
   at Microsoft.SharePoint.Administration.SPDatabase.Upgrade(Boolean recursively)
   at Microsoft.SharePoint.Administration.SPContentDatabase.<>c__DisplayClass2.<Upgrade>b__0()
   at Microsoft.SharePoint.SPSecurity.RunAsUser(SPUserToken userToken, Boolean bResetContext, WaitCallback code, Object param)
   at Microsoft.SharePoint.Administration.SPContentDatabase.Upgrade(Boolean recursively)
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively)
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively)
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively)
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively)
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse)
   at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

No other errors or other useful info appear in the log. These messages are logged directly prior to the error:

Successfully created security token service application
Creating service connection point for this farm ...
Successfully created service connection point using url 'https:///Topology/topology.svc'
Entering function TaskCommon.TraceToPsconfigLogAndUpgradeManagerLog
Not spawning off the upgrade job, but rather running upgrade synchronously by calling SPFarm.Local.Upgrade() for inplace upgrade Leaving function TaskCommon.TraceToPsconfigLogAndUpgradeManagerLog
Task upgrade has failed with an unknown exception

I tried all the steps as both Farm Admin and System Admin - no luck.

Any ideas what else could be wrong?

Answer 1

I had to view the Upgrade-*.log (not mentioned anywhere afaik) to notice that the process is obviously not executing past a specific content database. I had to detach this problematic content database through Central Admin (no idea why it was problematic). After this, I started getting SPUpdatedConcurrencyException, which can be resolved by running (on every front end):

stsadm -o setproperty -pn command-line-upgrade-running -pv No

followed by:

Psconfig.exe -cmd upgrade -inplace b2b -wait -force

I had to clear the configuration cache before this step as well.

Ultimately, to get the upgrade to complete, I had to perform steps described at these links, which other people encountering this problem may find helpful:

http://alstechtips.blogspot.com/2014/05/sharepoint-2013-product-configuration.html

https://techsps.wordpress.com/2013/11/26/psconfig-stuck-at-10-00/

http://nitrohn.blogspot.com/2014/06/psconfig-stuck-sharepoint-2013-server.html

http://www.sharepointgeoff.com/spupdatedconcurrencyexception-fix-when-trying-to-install-june-2011-cu-for-sharepoint-server/4

http://blogs.msdn.com/b/opal/archive/2010/04/18/track-sharepoint-2010-installations-by-service-connection-point-ad-marker.aspx (step 10 is not required)

A giant waste of time.