I tried looking into SQL*Loader. Apparently you should be able to right click a table > Import Data > next and there should be an option to generate SQL*Loader files.
Unfortunately, not only did the import wizard not open with my large .xlsx files, the SQL*Loader option was not even present for smaller .xlsx files, or even .xls for that matter.
In the end, I decided to convert my .xlsx files into .csv and import those instead. This worked for all but 4 rows of my 1.6 million, and gave me the insert statement for those 4, of which 2 worked when run with no additional modifications.