All plain FTP servers are inherently insecure as they authenticate using plain text over an unencrypted link. This means that anyone on the same network (in particular WiFi networks) can sniff the network traffic and easily extract the username and password used to authenticate.
You've already mentioned some of the recommended alternatives, SSH and SFTP. SSH provides the means to use SCP to transfer files securely. SFTP also uses the underlying transport provided by SSH. There is also FTPS which uses standard FTP over an SSL encrypted connection.
Have a look at this link for more information on the differences between the protocols.
In researching Go libraries for FTP I came across a mention that the github.com/jlaffaye/goftp library breaks on multi-line responses.
It's highly recommended to use a more secure protocol than plain FTP so you would be better served by looking at SCP/SFTP/FTPS solutions. Here's a Github Gist claiming to be an example of using SCP in Go. That could be a good starting point.