Your observations are correct. PHP's built-in webserver is usually slower than Apache+PHP, but for quick & dirty testing on your machine it's fine.
I prefer it over full stacks, because I don't like having Apache as a Service on my system which always starts (as XAMPP does by default) manually starting/restarting a Windows Service on the other hand can be quite annoying (compared to a simple php -S
). You also might have to change configs (e.g. when using vhosts), copy/symlink your project, maybe edit your /etc/hosts-file. All in all I think the built-in server is less hassle, than full stacks like WAMP.
I don't think @Areks concern weighs heavily against using the built-in server. If this really is a concern for you, you should account for different systems/configurations, e.g. by writing tests and using tools TravisCI, Vagrant and/or others. If you develop for a specific system you probably have a staging environment (as similar to the production env as possible) anyway.