Writing to file with cython.parallel.parallel and nogil

Question

I'm getting my feet wet with Cython. Consider me a total noob with it. Writing to a very large file is a major bottleneck in my code, so I thought I'd look into parallelism, but I haven't been able to find anything useful on writing files with parallelism.

Is it even possible to use a

with nogil, parallel():

statement with writing to a file? I get compilation errors when trying to write a string:

Constructing Python tuple not allowed without gil

Answer 1

You can't use any Python functions or objects within a nogil block. If you want to do file IO with nogil, you'll have to do it with C. This blog post can probably help you. Specifically, you can cimport the usual C functions from stdio. These functions should be usable within nogil blocks. The below is from the blog post to which I linked above:

from libc.stdio cimport *

cdef extern from "stdio.h":
    FILE *fopen(const char *, const char *)
    int fclose(FILE *)
    ssize_t getline(char **, size_t *, FILE *)