I need to restrict direct access only to directories and sub directories. I need to access whatever the flies inside the directories and sub directories.
What I meant was to prohibit directory listing. And show to 403 instead
You can try following rule:
# disable directory listing
Options -Indexes