You need to obtain IShellBrowser and use QueryActiveShellView to get the shell view, then QI for IFolderView on the shell view and call IFolderView::SetCurrentViewMode.
On the vista style file dialog (that is, if you don't disable the automatic upgrade in the constructor of CFileDialog, and you are running on Vista+), you can get IShellBrowser from the IFileDialog object via its IServiceProvider interface (QueryService with SID_STopLevelBrowser). On the Windows 2000 style file dialog, you can send an undocumented message WM_GETISHELLBROWSER to the file dialog to get its IShellBrowser interface. An example can be found at https://jiangsheng.net/2021/06/16/better-late-than-never/.