Yes, you should sign your app, even if you are not submitting it to the App Store. See the Developer ID and Gatekeeper guide for how to do this. Yes, this is why you are seeing the warning.
By default, System Preferences > Security & Privacy
is set to allow applications from Mac App Store and identified developers
. By signing your app, you are an identified developer
. Setting it to Anywhere
would allow the user to install an unsigned app (as well as make them more vulnerable to malware, viruses, etc.)