Starting with Outlook 2013, Microsoft introduced new performance criteria for add-ins. For more information see http://msdn.microsoft.com/en-us/library/office/jj228679.aspx#ol15WhatsNew_AddinDisabling
In your case, Outlook is loading the .Net framework and is considering the cost as part of your add-in. Native COM add-ins tend to load faster than .Net add-ins.
As the article points out (in the section "System Administrator control over add-ins", there is a Group Policy setting that allows you to specify which add-ins are always enabled, always disabled (blocked), or configurable by the user.
Though not recommended, as Outlook does when "Always enable this Add-In" is selected (as described here), you could directly add your add-in to the following registry key.
HKCU\Software\Microsoft\Office\15.0\Outlook\Resiliency\DoNotDisableAddinList
For improving the performance of VTSO based add-in, see Performance Improvements Coming Soon to a Service Pack Near You
For changing when an add-in is loaded, see Delay-loading the CLR in Office Add-ins.