The icon inside the exe file is set by cx_Freeze when you freeze it. You can set the icon
parameter to Executable:
executables = [Executable("guifoo.py", base=base, icon="my_icon.ico")]
I think the shortcut icon is set by Inno setup, although it may copy the exe icon by default.
EDIT: It turns out it was also necessary to set the window icon within the program - in Tkinter, this is done like this:
root.iconbitmap(default='myicon.ico')