Are you sure you have your app.config setup properly? I'm using EF6.1 with System.Data.SQLite v1.0.92.0 using the following app.config:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="entityFramework" type="System.Data.Entity.Internal.ConfigFile.EntityFrameworkSection, EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false" />
</configSections>
<system.data>
<DbProviderFactories>
<remove invariant="System.Data.SQLite" />
<add name="SQLite Data Provider" invariant="System.Data.SQLite" description=".Net Framework Data Provider for SQLite" type="System.Data.SQLite.SQLiteFactory, System.Data.SQLite" />
<remove invariant="System.Data.SQLite.EF6" />
<add name="SQLite Data Provider (Entity Framework 6)" invariant="System.Data.SQLite.EF6" description=".Net Framework Data Provider for SQLite (Entity Framework 6)" type="System.Data.SQLite.EF6.SQLiteProviderFactory, System.Data.SQLite.EF6" />
</DbProviderFactories>
</system.data>
<entityFramework>
<providers>
<provider invariantName="System.Data.SQLite" type="System.Data.SQLite.EF6.SQLiteProviderServices, System.Data.SQLite.EF6" />
</providers>
</entityFramework>
</configuration>
This will need to be in the app.config for the executing program not the app.config for the assembly which contains your DbContext. Alternatively you could use code based configuration as I answered here which you can place in the same assembly as your DbContext.
And also yes if your DbContext is in a different assembly then that assembly will need some code referencing SQLite otherwise the compiler won't include the references to the required SQLite assemblies. You've provided an incorrect reference above to SqlServer rather than SQLite which won't work.