Yes you should make the Token endpoint as Secure.
In the Setup.Auth.cs
file under the OAuthurizationServerOptions you can specify to be Token end point requires SSL or not.
OAuthOptions = new OAuthAuthorizationServerOptions
{
TokenEndpointPath = new PathString("/Token"),
Provider = new ApplicationOAuthProvider(PublicClientId),
AuthorizeEndpointPath = new PathString("/api/Account/ExternalLogin"),
AccessTokenExpireTimeSpan = TimeSpan.FromMinutes(20),
AllowInsecureHttp = false
};
The AllowInsecureHttp
will force the url to be SSL or not.