How to properly set Principle to retrieve that during request life-time?

Question

The problem here is that:

You're not aware of the controllers lifecycle. YES, they DO have one, and it bound to the request.
You're using a custom authorization/authentication implementation

For the request (and controller) lifecycle, take a look into this sample:

Request -> new Controller() -> controller.Action() -> Response -> Dispose()

This being said, it's no wonder that setting the Thread.CurrentPrincipal or even the HttpContext.Current.User inside your TokenController won't work: the request already ended and the thread is now disposed. When you request another action/method, it's another entire context, with new threads and a new User/Identity/Principal object!

Imagine if you have 10 users. Which principal/identity should be returned by User.Identity if it was not bound to the current request's context? That's why...

So, you're better shot here would be to implement an ActionFilter or an AuthenticationFilter to be called BEFORE your controller, within the same request context, to correctly set the identity/principal so it's available for any controller, anytime.

We've created a custom auth framework here once. It's not that hard. Here follows some samples:

public MyCustomAuthenticationFilter : FilterAttribute, IActionFilter
{
   public void OnActionExecuted(ActionExecutedContext filterContext)
   {
      // You may set the user here.
   }

   public void OnActionExecuting(ActionExecutingContext filterContext)
   {
      // This will be called after the action has executed. Not that useful right now...
   }

   public class Override : FilterAttribute, System.Web.Mvc.Filters.IOverrideFilter
   { 
      public Type FiltersToOverride { get { return typeof(MyCustomAuthenticationFilter); } }
   }
}

Than, you may simply register it as a global filter:

GlobalFilters.Filters.Add(new MyCustomAuthenticationFilter());

Wherever you DON'T want it to execute, you may use a filter override:

[MyCustomAuthenticationFilter.Override]
public ActionResult Something()

But you'll need your custom filter provider, which can resolve all these (and other) overrides, similar to:

public class MyCustomFilterProvider : System.Web.Mvc.IFilterProvider
{
    public IEnumerable<System.Web.Mvc.Filter> GetFilters(System.Web.Mvc.ControllerContext controllerContext, System.Web.Mvc.ActionDescriptor actionDescriptor)
    {
        var _globalFilters = System.Web.Mvc.GlobalFilters.Filters;
        var _controllerFilters = controllerContext.Controller.GetType().GetCustomAttributes(true).OfType<FilterAttribute>();
        var _actionFilters = actionDescriptor.GetCustomAttributes(true).OfType<FilterAttribute>();

        var _all = _globalFilters.Union(_controllerFilters).Union(_actionFilters).ToList();
        var _overrides = _all.Where(_f => _f.Instance.GetType().GetInterfaces().Any(_i => _i.Equals(typeof(System.Web.Mvc.Filters.IOverrideFilter)))).ToArray();

        foreach(var _override in _overrides)
        {
            var _found = _all.Where(_f => _f.Instance.GetType().Equals(((System.Web.Mvc.Filters.IOverrideFilter)_override.Instance).FiltersToOverride)).ToArray();
            foreach (var _item in _found) _all.Remove(_item);
        }

        return _all.Except(_overrides);
    }
}

This provider should be registered to work (of course), so:

FilterProviders.Providers.Add(new MyCustomFilterProvider());

And I think that's all. Should not get in your way anymore.

You may, however, avoid ALL this and simply use the new AspNet.Identity/Katana/Owin (you don't have to use EF if that's the issue) and store everything you need as claims, making it really useful and easy. It works with bearer/token authentication/authorization too!