These rules should meet both requirements. But allows you to change the category and subcat as long as the normal URLs don't change and format
is always a key in the query string.
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)/(.+)/(.+)$ /$1/$2/?format=$3 [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)/(.+)$ /$1/?format=$2 [L]
Then you should be able to use these URL types.
http://example.com/category/feed
http://example.com/category/sub_category/feed