If you want to remove the /blog/category/ portion from your category URL's, first install the WP No Category Base plugin in WordPress, which will remove the /category/ from your URL.
Next, to remove the /blog/ portion from your URL's, you will need Root.
To configure root, first make sure you have the latest version of Magento WordPress Integration installed and then install Root. Login to your Magento Admin and select WordPress > Settings > Integration and set 'Integrate at Root' to 'Yes' and 'Replace Homepage with WordPress' to 'No' (screenshot below).
Next, login to your WordPress Admin and create a page called 'Blog' with a page slug of 'blog'. Set Appearance > Customize > Static Front Page to 'A static page' and set the front page to be empty and your posts page to be the page named Blog, which you just created (screenshot below). Then hit save.
Your blog homepage will now be visible from the /blog/ URL just as before, however all of your other blog URL's will be available from the root. If you want your blog posts to be available from /blog/, go to your permalink settings in WordPress and add /blog/ to the start of your custom permalink.