Regular users cannot use the Directory API, you'll need to authenticate as an admin user to make Directory API calls.
Depending on your needs though for accessing all users, you may be able to get by with requesting access to the user's Contacts scope and grabbing a copy of the full Global Address List which contains information on all non-hidden domain users as well as non-hidden groups and shared contacts.