Process on linux ignoring resource limits

Question

I had asked some questions about developing an online judge sometime ago on stackoverflow and I found quite a few good answers to that. I've started working on developing one and I seem to have run into a major flaw in my code.

The user submitted source will be compiled on the server. This is done by exec()ing gcc in a forked process. Now I set a resource limit on the CPU time and on exceeding that, a SIGXCPU signal is sent to the process. All is fine till now. But suppose someone writes a malicious code that handles the SIGXCPU code itself, it would then continue running on the server and maybe open up a way for someone to take on remote control of the server.

So what am I missing here? There must be someway this can be prevented.

The basic prototype of the compiling module goes like this:

int main()
{
    int pid;
    int rv;

    if (!( pid=fork() ))
    {
        struct rlimit limit;
        getrlimit(RLIMIT_CPU, &limit);

        limit.rlim_cur = 1;

        setrlimit(RLIMIT_CPU, &limit);

       //execl() with gcc and source file name
    }
    else if(pid)
    {
        wait(&rv);
    }
    else
        printf("Error forking\n");

    return 0;
}

and if the source file contains something like

void handler(int signum)
{
    if (signum == SIGXCPU)
        printf("Caught SIGXCPU signal\n");
}

int main()
{
signal(SIGXCPU, handler);
while(1);
return 0;
}

...this is big trouble

Answer 1

On linux, specifically, a user could do what you say. But linux will send sigkill to the process if the hard limit is reached(as opposed to the soft limit you've set), and that will terminate the process.

(Remember though, you really really need to run your stuff in a chrooted environment)

Answer 2

Wow. Having thought about this daunting undertaking for all of about 7 minutes I apologize in advance for anything stupid I am about to say.

Is this going to be something like UVA Judge?

If the goal is to allow relatively simple programs to run without allowing malicious users destroying your system then it seems you need to be more proactive than this or you will be patching holes until the end of time.

At a minimum I think you would need to strip out user header files and substitute one of your own that contains the minimum functionality. Disallow assembler. Use a modified stdlib and/or kernel that no-ops or kills the process on any attempted syscall(), etc.

There is an awful lot to consider here.