Question

I've been wracking my brain over this for the last few days now and haven't been able to find much in the way of help for my specific issue. Here's my setup.

Windows 7
IIS 7.5
Helicon Zoo 3.1.98.508
Django 1.6.3
Python 2.7.3

I should state that I'm new to Django and Helicon Zoo.

What's happening is that while I seem to have no issue uploading a file under the default of 2.5mb, I cannot get chunks to upload correctly for anything over this amount. While I'm sure I could up this limit in my settings file, I'd really rather not as for my purpose I could see needing to upload larger files than what I would really want being handled by memory.

What I'm seeing when I attempt this is a rather uninformative Error 500 from Helicon Zoo about some missing headers

Worker Status
The process was created
Windows error
The operation completed successfully. (ERROR CODE: 0)
Internal module error
message: HTTP-headers - are expected  
type: ZooException 
file: Jobs\JobFastCgi.cpp 
line: 391 
version: 3.1.98.508 
STDERR
Empty stderr

At first, I thought maybe it was something to do with the temp folder so I set

FILE_UPLOAD_TEMP_DIR = [os.path.join(BASE_DIR, 'temp')]

in my settings file, but it didn't seem to do anything useful.

I then discovered how to enable logging for django which really opened things up to me. I know now where the error is coming from.. but I have no clue what I'm supposed to do to get around it..

[29/Apr/2014 11:37:00] ERROR [django.request:226] Internal Server Error: /upload/
Traceback (most recent call last):
  File "E:\mysite\venv\lib\site-packages\django\core\handlers\base.py", line 107, in get_response
    response = middleware_method(request, callback, callback_args, callback_kwargs)
  File "E:\mysite\venv\lib\site-packages\django\middleware\csrf.py", line 170, in process_view
    request_csrf_token = request.POST.get('csrfmiddlewaretoken', '')
  File "E:\mysite\venv\lib\site-packages\django\core\handlers\wsgi.py", line 146, in _get_post
    self._load_post_and_files()
  File "E:\mysite\venv\lib\site-packages\django\http\request.py", line 215, in _load_post_and_files
    self._post, self._files = self.parse_file_upload(self.META, data)
  File "E:\mysite\venv\lib\site-packages\django\http\request.py", line 180, in parse_file_upload
    return parser.parse()
  File "E:\mysite\venv\lib\site-packages\django\http\multipartparser.py", line 197, in parse
    charset)
  File "E:\mysite\venv\lib\site-packages\django\core\files\uploadhandler.py", line 135, in new_file
    self.file = TemporaryUploadedFile(self.file_name, self.content_type, 0, self.charset)
  File "E:\mysite\venv\lib\site-packages\django\core\files\uploadedfile.py", line 61, in __init__
    dir=settings.FILE_UPLOAD_TEMP_DIR)
  File "E:\mysite\venv\lib\site-packages\django\core\files\temp.py", line 27, in __init__
    dir=dir)
  File "C:\python27\lib\tempfile.py", line 300, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags)
  File "C:\python27\lib\tempfile.py", line 235, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
TypeError: coercing to Unicode: need string or buffer, list found

What does this mean? Are my chunks coming in as a list object somehow and I don't realize it?

Here's my code for the app, it's included in my Blog app I made since it's my main app and it seemed simpler than creating an app all it's own. Was that a mistake?

forms.py

from django import forms

class UploadFileForm(forms.Form):
    title = forms.CharField(max_length=50)
    file = forms.FileField()

views.py

from django.shortcuts import render, render_to_response
from django.http import HttpResponse, Http404, HttpResponseRedirect
from django.template import RequestContext, loader
from blog.forms import UploadFileForm
from blog.models import Blog
from blog.uploads import handle_uploaded_file


def index(request):
    latest_blog_list = Blog.objects.order_by('-pub_date')[:5]
    template = loader.get_template('blog/index.html')
    context = RequestContext(request, {
        'latest_blog_list': latest_blog_list,
        })
    return HttpResponse(template.render(context))

def detail(request, blog_id):
    try:
        blog = Blog.objects.get(pk=blog_id)
    except:
        raise Http404
    return render(request, 'blog/detail.html', {'blog': blog})

def upload_file(request):
    if request.method == 'POST':
        form = UploadFileForm(request.POST, request.FILES)
        if form.is_valid():
            handle_uploaded_file(request.FILES['file'])
            #form.save()
            return HttpResponseRedirect('/upload/')
    else:
        form = UploadFileForm()
    return render_to_response('upload/upload.html', {'form': form}, context_instance=RequestContext(request))

uploads.py

import os

saveDir = 'E:\\uploads\\'

def handle_uploaded_file(f):
    #logging.debug('upload_here')
    if f:
        destination = open(saveDir + f.name, 'wb+')
        for chunk in f.chunks():
            destination.write(chunk)
        destination.close()

app urls.py

from django.conf.urls import patterns, url
from blog import views

urlpatterns = patterns('',
                       url(r'^$', views.index, name='index'),
                       url(r'^(?P<blog_id>\d+)/$', views.detail, name='detail'),
                       url(r'^upload/$', views.upload_file, name='upload'),
)

project urls.py

from django.conf.urls import patterns, include, url
from django.conf import settings
from django.conf.urls.static import static
from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
    # Examples:
    # url(r'^$', 'project.views.home', name='home'),
    # url(r'^blog/', include('blog.urls')),
    url(r'^/?', include('blog.urls', namespace="blog")),
    url(r'^admin/', include(admin.site.urls)),
    url(r'^upload/', include('blog.urls', namespace="upload")),
)

I should also note that portions of these large files are actually appearing in my upload location, but nothing more than 1 meg in size. It's like the process stops before it's actually done for some reason. I then have to stop and restart IIS to get the site to even function again.

Was it helpful?

Solution

So there were a couple of different problems here. First off, I discovered that the

TypeError: coercing to Unicode: need string or buffer, list found

error was not because of the object being sent by my code, but rather due to the setting

FILE_UPLOAD_TEMP_DIR = [os.path.join(BASE_DIR, 'temp')]

needing to instead be

FILE_UPLOAD_TEMP_DIR = os.path.join(BASE_DIR, 'temp')

THAT is what was being sent as a list when it shouldn't have been, simple mistake in trying to replicate other fields within the django project settings file. I should have noticed the difference between DIRS and DIR, of course denoting a singular reference.


Secondly, the error continued after resolving that matter. After a couple days of beating my head trying to find maybe an IIS setting that could be truncating my uploads to 1MB in size, or potentially a django setting I was overlooking, and even testing a prebuilt app, only to still hit the same brick wall, it finally dawned on me when reading through the Helicon Zoo error report's list of configuration variables, that I noticed something...

postBuffer = 1024

Son of a... So after a little research I discovered I could modify this setting by using an environment variable inside of my Helicon Zoo web.config.

POST_BUFFER = "4096"

Viola.. I can suddenly upload anything over 1mb in size... anything over 4mb is truncated and I get an error!

So I finally found the issue, but its completely surprising to me that I can't find any sort of documentation regarding this as being an issue. Surely I'm not the only person whose come across this as it's a default setting! I wish I could remove it altogether as even being a limitation since django handles this just fine itself, I really don't need Helicon doing it for me. Unfortunately it looks like if I set the value to 0 then no uploads will work at all. So for now I'm going to set this value to something arbitrarily high just because frankly it's not needed and doesn't help me in any way.

UPDATE: Thanks to rukeba's comments above, his solution was the final answer to this issue as the Twisted engine does not have this problem.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top