Why are some files are not being added do my zipfile

Question

I have the following code:

with ZipFile('deploy.zip', 'w') as deploy:
    if os.path.isfile(artifact.source):
        deploy.write(artifact.source, artifact.target)
    else:
        for base, dirs, files in os.walk(artifact.source):
            for file_ in files:
                source = os.path.join(base, file_)
                target = os.path.join(base[base.index(artifact.target):], file_)
                deploy.write(source, target)

when this code finish, only the files that match when the artifact.source is a file are added to the deploy.zip. And in some cases artifact.source will be a directory (i have tested this case too) ad the for part will be executed.

The result of the following lines are valid and source exists for every iteration:

source = os.path.join(base, file_)
target = os.path.join(base[base.index(artifact.target):], file_)

Here the full code that i'm working: https://gist.github.com/khaoz/9b04d87b0900fba780f0 Set config.project_root to something like "c:\temp" and remove the import config line. OBS: I'm a Python newbie, so ignore some crap code that you will see :P

And here a example of my csv file: https://gist.github.com/khaoz/e9a59390f415f22d46db

What i'm doing wrong ?

Answer 1

I discovered the problem. Sometimes, sleep is the best solution form some problems

I was doing:

for artifact in artifacts:
    if not artifact.name in contents:
        contents.append(artifact.name)

    with ZipFile('deploy.zip', 'w') as deploy:
        if os.path.isfile(artifact.source):
            deploy.write(artifact.source, artifact.target)
        else:
            for base, dirs, files in os.walk(artifact.source):
                for file_ in files:
                    source = os.path.join(base, file_)
                    target = os.path.join(base[base.index(artifact.target):], file_)
                    deploy.write(source, target)

But for every iteration in the artifacts close and open a new deploy.zip file.

The right way to do this is:

with ZipFile('deploy.zip', 'w') as deploy:
    for artifact in artifacts:
        if not artifact.name in contents:
            contents.append(artifact.name)

        if os.path.isfile(artifact.source):
            deploy.write(artifact.source, artifact.target)
        else:
            for base, dirs, files in os.walk(artifact.source):
                for file_ in files:
                    source = os.path.join(base, file_)
                    target = os.path.join(base[base.index(artifact.target):], file_)
                    deploy.write(source, target)

And everything works as expected.

Thank you very much for everyone that tried to help. Next time i will post the full source code or at last, some lines more. :)

Answer 2

JUST FYI

My interpretation of what you did, this seems to work.

from zipfile import ZipFile
from collections import namedtuple
import os

Artifact =  namedtuple('Artifact', ['source', 'target'])
artifact =  Artifact(source="Mongodb", target="g")

with ZipFile('deploy.zip', 'w') as deploy:
    if os.path.isfile(artifact.source):
        print "F"
        print "\n", artifact.source
        print "\n", artifact.target
        deploy.write(artifact.source, artifact.target)
    else:
        for base, dirs, files in os.walk(artifact.source):
            for file_ in files:
                print "base", base, file_
                source = os.path.join(base, file_)
                target = os.path.join(base[base.index(artifact.target):], file_)
                print "f"
                print "\t", source
                print "\t", target
                deploy.write(source, target)

unzip -l deploy.zip | tail

     2591  01-09-13 21:26   godb/Sortif/scratch.py
     2010  01-15-13 20:20   godb/Sortif/sortif_model.py
     2495  01-15-13 20:22   godb/Sortif/sortif_model.pyc
      161  01-15-13 20:45   godb/Sortif/sortif_scratch.py
        0  01-08-13 12:05   godb/Sortif/sortif/__init__.py
        0  01-08-13 12:05   godb/Sortif/sortif/models/__init__.py
     1408  01-21-13 18:05   godb/ZeroMQ/client.py
     3044  01-21-13 17:51   godb/ZeroMQ/controller.py
 --------                   -------
 11137644                   967 files

I'm unsure what you're trying to achieve with base[base.index(artifact.target):] are you wanting to change the prefix ? Because running it on my Mongodb directory target had to appear in the directory of the file base.

I'm unsure how you expect to drive the code as it seems that artifact.source is a constant. So first time around it finds a file and it will never do the part where it's looking for directory.

Shouldn't it be

with ZipFile('deploy.zip', 'w') as deploy:
    for artifact in articats:
        if os.path.isfile(artifact.source):
            ...