I think the issue is with a custom theme. Try doing this steps:
- Remove all active widgets
- Uninstall all plugins
- Check loading time - if not improved it's an issue with custom theme.
One more thing to do (to check if it's not the host issues) - activate default wordpress theme and check average loading speed, if speed is not improved try to configure your server correctly or change hosting plan or hosting provider.