You don't need system()
. You can use fork/execlp
if(!fork())
execlp("gedit", "gedit", filename, NULL);
The version of gedit that's on my laptop uses a new tab by default if there's a running instance already, but I'm not sure about other versions.