If you really want to use MVVM strictly, your custom controls can't be a part of main ViewModels. That's why you have to separate them in a (Control)Library and add to your project as a reference. (The main goal of MVVM: Graphics experts can work on the frontend layer(in this case view using xaml) independently meanwhile programmers can work on the backend layer(in this case model). And Programmers can create a connection between these layers (viewmodel). That's why strict MVVM doesn't allow code-behind.) So back to your question, "Are they in a separate ControlLibrary?" -Yes, it can be a solution. But there's the most important thing: If you place custom elements in a control library you can SHARE your work with multiple WPF applications(easily).
There's a book: http://www.amazon.com/Pro-WPF-2010-Presentation-Foundation/dp/1430272058 What you need is in Chapter 18 - Custom Elements. There's a long 'explain' why you should use control library and what about special cases etc... (sorry for my english)