You probably shouldn't worry about it.
If your ASP.NET MVC site is like a normal site, and has out-of-process call, like
- accessing a database
- calling web services
- accessing the file system
- etc.
then the majority (95+ %) of the page request time is going to be used there.
In general, DI Containers like NInject are fast; they are reusable libraries, and their creators have typically worked on making them as fast as possible. In fact, they don't do anything that MVC's internal DependencyResolver doesn't already do.
For more on this topic, read my (container-agnostic) article Compose object graphs with confidence, or view my talk Big Object Graphs Up Front.
You should also keep in mind Knuth's words, that premature optimization is the root of all evil. If you're worried about it: measure.
That's what I did, and what I report on, in my Big Object Graphs Up Front talk.