No, your design is not vulnerable to XSS attacks. At no point do you use any user-supplied content in generating your web pages themselves.
The content passes through our site as a black box instead; from browser to your filesystem, then back to other HTTP clients that may choose to download the content again.
The filename you control entirely, you never read the contents of the file, you never take any of the file contents and put them into generated HTML content.
The only problem that may exist is that someone uploads a file that is not XML or JSON, really, but a file that exploits a vulnerability in whatever the downloader uses to parse the file. That's not a XSS attack however.