You can refer to this question:
Why NuGet adds app.config with assemblyBinding to LIBRARY projects during a NuGet package update?
I think the answer to the above question amply answers your query.
Still, if you want the summary, you are actually right. You just need one .config file in a single application. Per-assembly .config files are used in tests when you only are using/referencing the respective assembly. So, if you are deploying the application, you do not need the per-assembly .config files. The application wide .config file would suffice.