I wasn't able to stop haveged without an error being thrown.
I also moved to using checkinstall when installing things from source, it will package up all of the files into a package and will install it that way, which makes it easier to remove all associated files.
I removed all the files that the source version of haveged used, but I think the main culprit was /etc/init.d/haveged
Then "sudo apt-get upgrade" worked with no problems