Not all channels are enabled for custom thumbnails; I suspect that if you're getting an error, it might be because you're attempting to set a custom thumbnail for a video in a channel that isn't enabled. There's more info at https://support.google.com/youtube/answer/72431?hl=en
Could you follow the steps in that help article and see whether you have an option in the web UI to set a custom thumbnail? If you don't, then your channel isn't enabled.
Getting back a HTTP 500 response is obviously unhelpful, though, and makes it difficult to confirm that that's what's going on. We can follow up with the engineering team about that to get a helpful error returned once we confirm whether your channel is enabled or not.