You cannot set limits by running a shell, because that limit will only affect the shell process you created (and not the tcl
process itself).
You need to find out how to call the setrlimit(2) syscall directly. Perhaps tclx might help.
Take several hours to read Advanced Linux Programming.
You are misunderstanding some fundamental Unix concepts, notably processes.