What about generating a private key and display it to the user?
I use this snippet as part of the entrypoint script for an image:
KEYGEN=/usr/bin/ssh-keygen
KEYFILE=/root/.ssh/id_rsa
if [ ! -f $KEYFILE ]; then
$KEYGEN -q -t rsa -N "" -f $KEYFILE
cat $KEYFILE.pub >> /root/.ssh/authorized_keys
fi
echo "== Use this private key to log in =="
cat $KEYFILE