Your rules aren't correct actually. As RewriteCond %{REQUEST_FILENAME} -d
means an existing directory and /aaa/bbb
is not a directory.
Try this code:
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^(sub)/?$ /sub/try.php [L]
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(sub/bbb)/?$ /sub/foo.php [L]