os.path.join(p1,p2) omits slashes of p1

Question

I am passtime programmer so please be gentle. Now for the actual problem one of my users is encountering this strange behaviour where os.path.join(p1,p2) returns a relative path with all the slashes omitted from p1. Like so(pretending this is done within the python cmd line interpreter):

>>import os
>>p1 = "/Some/Path/Tosmth"
>>p2 = "file.ext"
>>print os.path.join(p1,p2)`

Then the output is:

>>"SomePathTosmth/file.ext"

Just before the join operation I checked the content of p1 and p2 and it is just what I would expect. Here is the actual implementation in question with some additional debugging code:

   def __moveMovie(self, src, dst, folder, file_name):
    try:
        self.logDebug('__moveMovie(): src=%s | dst=%s | folder=%s | file_name=%s' % (src, dst, folder, file_name))
        dest = save_path(dst)
        file_name = save_path(file_name)
        if self.getConfig("subfolder") is True:
            dest = os.path.join(dst,folder)
            os.mkdir(Utils().encode(dest))
    except OSError, e:
        if e.args[0] == 17:
            self.logDebug(u'Cannot create folder "%s". It already exists.' % os.path.join(dest))
    try:
        full_dst = Utils().encode(os.path.join(dest,file_name))
        self.logDebug('var "full_dst" w/o encode: %s' % os.path.join(dest, file_name))
        self.logDebug('var "full_dst" w/ encode: %s' % Utils().encode(os.path.join(dest,file_name)))
        if os.path.exists(full_dst):
            pass
        shutil.move(src, full_dst)
        self.logInfo(u'Movie "%s" moved to "%s"' % (os.path.split(src)[1], os.path.join(dest,file_name)))
        self.__movie_queue.task_done()
    except OSError, e:
        if e.args[0] == 21:
            self.logDebug(u'Cannot move "%s" to "%s". "%s" is a directory.' % (os.path.split(src)[1],
                                                                               os.path.join(dest, file_name),
                                                                               os.path.join(dest, file_name)))
            self.__movie_queue.task_done()

This is the log for that code:

05.06.2013 17:29:12 DEBUG     MovieMover: __moveMovie(): src=/var/raid/Daten/Neu/abgezockt.tc.72-ps/Voll.Abgezockt.2013.UNRATED.GERMAN.AC3LD.5.1.DL.720p.BluRay.x264-DerSchuft.mkv | dst=/var/raid/Daten/Filme | folder=Voll.Abgezockt.2013.UNRATED.GERMAN.AC3LD.5.1.DL.720p.BluRay.x264-DerSchuft | file_name=Voll Abgezockt.mkv
05.06.2013 17:29:12 DEBUG     MovieMover: var "full_dst" w/o encode: varraidDatenFilme/Voll Abgezockt.mkv
05.06.2013 17:29:12 DEBUG     MovieMover: var "full_dst" w/ encode: varraidDatenFilme/Voll Abgezockt.mkv
05.06.2013 17:29:12 DEBUG     MovieMover: __moveMovie(): src=/var/raid/Daten/Neu/abgezockt.tc.72-ps/.AppleDouble/Voll.Abgezockt.2013.UNRATED.GERMAN.AC3LD.5.1.DL.720p.BluRay.x264-DerSchuft.mkv | dst=/var/raid/Daten/Filme | folder=Voll.Abgezockt.2013.UNRATED.GERMAN.AC3LD.5.1.DL.720p.BluRay.x264-DerSchuft | file_name=Voll Abgezockt.mkv
05.06.2013 17:29:12 DEBUG     MovieMover: var "full_dst" w/o encode: varraidDatenFilme/Voll Abgezockt.mkv
05.06.2013 17:29:12 DEBUG     MovieMover: var "full_dst" w/ encode: varraidDatenFilme/Voll Abgezockt.mkv
Exception in thread Thread-1072:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/var/local/pyload/userplugins/hooks/MovieMover.py", line 360, in __getMvQueue
    self.__moveMovie(src, dst, movie.folder_name, movie.file_name)
  File "/var/local/pyload/userplugins/hooks/MovieMover.py", line 382, in __moveMovie
    shutil.move(src, full_dst)
  File "/usr/lib/python2.7/shutil.py", line 301, in move
    copy2(src, real_dst)
  File "/usr/lib/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: u'varraidDatenFilme/Voll Abgezockt.mkv'

Exception in thread Thread-1074:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/var/local/pyload/userplugins/hooks/MovieMover.py", line 360, in __getMvQueue
    self.__moveMovie(src, dst, movie.folder_name, movie.file_name)
  File "/var/local/pyload/userplugins/hooks/MovieMover.py", line 382, in __moveMovie
    shutil.move(src, full_dst)
  File "/usr/lib/python2.7/shutil.py", line 301, in move
    copy2(src, real_dst)
  File "/usr/lib/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: u'varraidDatenFilme/Voll Abgezockt.mkv'

I am at a total loss since the encoding function enclosing the os.path.join() function is not the cause and when he does the os.path.join() with his python cmd line interpreter the result is correct. I am writing in 2.5 while he is executing the code on 2.7.2 if thats of any interest. I hope someone is able to shed some light on this. Thank you!

Since it appears I can't use code formatting in the comments I'll answer those here:

This is the save_path() function:

def save_path(name):
    #remove some chars
    if os.name == 'nt':
        return remove_chars(name, '/\\?%*:|"<>')
    else:
        return remove_chars(name, '/\\"')

This now I must add is not my code. I am writing for an Open Source Project called pyLoad. Just want to make sure I am not claiming anything to be my code when it actually is not.

EDIT: Brendan Long's answer seems to be accurate. In my development environment self.getConfig("subfolder") was on/True while it appears it is not for my user. With that I could successfully reproduce the bug. I'll release that fix and let the user in question confirm it works but thus far it all points to save_path being the culprit. Also feel a bit like an idiot for overlooking the obvious, really. Anyway thanks and I'll get back to you with the result.

Answer 1

Looking at save_path:

if os.name == 'nt':
    return remove_chars(name, '/\\?%*:|"<>')
else:
    return remove_chars(name, '/\\"')

This function removes /'s (and a couple other things that have special meaning in file names). So.. that's why the slashes are being removed.

I'm guessing this function is meant to clean up the filenames of user-uploaded files (so they can't upload a file named ../../../etc/passwd). If that's not necessary, then the solution is to just leave out the call to this function.