Memory leak in a Tcl wrapper

Question

I read all I could find about memory management in the Tcl API, but haven't been able to solve my problem so far. I wrote a Tcl extension to access an existing application. It works, except for a serious issue: memory leak.

I tried to reproduce the problem with minimal code, which you can find at the end of the post. The extension defines a new command, recordings, in namespace vtcl. The recordings command creates a list of 10000 elements, each element being a new command. Each command has data attached to it, which is the name of a recording. The name subcommand of each command returns the name of the recording.

I run the following Tcl code with tclsh to reproduce the problem:

load libvtcl.so
for {set ii 0} {$ii < 1000} {incr ii} {
  set recs [vtcl::recordings]
  foreach r $recs {rename $r ""}
}

The line foreach r $recs {rename $r ""} deletes all the commands at each iteration, which frees the memory of the piece of data attached to each command (I can see that in gdb). I can also see in gdb that the reference count of variable recs goes to 0 at each iteration so that the contents of the list is freed. Nonetheless, I see the memory of the process running tclsh going up at each iteration.

I have no more idea what else I could try. Help will be greatly appreciated.

#include <stdio.h>
#include <string.h>
#include <tcl.h>

static void DecrementRefCount(ClientData cd);
static int ListRecordingsCmd(ClientData cd, Tcl_Interp *interp, int objc,
                             Tcl_Obj *CONST objv[]);
static int RecordingCmd(ClientData cd, Tcl_Interp *interp, int objc,
                        Tcl_Obj *CONST objv[]);

static void
DecrementRefCount(ClientData cd)
{
  Tcl_Obj *obj = (Tcl_Obj *) cd;
  Tcl_DecrRefCount(obj);
  return;
}

static int
ListRecordingsCmd(ClientData cd, Tcl_Interp *interp, int objc,
                  Tcl_Obj *CONST objv[])
{
  char name_buf[20];
  Tcl_Obj *rec_list = Tcl_NewListObj(0, NULL);

  for (int ii = 0; ii < 10000; ii++)
    {
      static int obj_id = 0;
      Tcl_Obj *cmd;
      Tcl_Obj *rec_name;

      cmd = Tcl_NewStringObj ("rec", -1);
      Tcl_AppendObjToObj (cmd, Tcl_NewIntObj (obj_id++));

      rec_name = Tcl_NewStringObj ("DM", -1);
      snprintf(name_buf, sizeof(name_buf), "%04d", ii);
      Tcl_AppendStringsToObj(rec_name, name_buf, (char *) NULL);
      Tcl_IncrRefCount(rec_name);

      Tcl_CreateObjCommand (interp, Tcl_GetString (cmd), RecordingCmd,
                            (ClientData) rec_name, DecrementRefCount);
      Tcl_ListObjAppendElement (interp, rec_list, cmd);
    }

  Tcl_SetObjResult (interp, rec_list);

  return TCL_OK;
}

static int
RecordingCmd(ClientData cd, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[])
{
  Tcl_Obj *rec_name = (Tcl_Obj *)cd;
  char *subcmd;

  subcmd = Tcl_GetString (objv[1]);
  if (strcmp (subcmd, "name") == 0)
    {
      Tcl_SetObjResult (interp, rec_name);
    } 

  else
    {
      Tcl_Obj *result = Tcl_NewStringObj ("", 0);
      Tcl_AppendStringsToObj (result,
                              "bad command \"",
                              Tcl_GetString (objv[1]),
                              "\"",
                              (char *) NULL);
      Tcl_SetObjResult (interp, result);
      return TCL_ERROR;
    }

  return TCL_OK;
}

int
Vtcl_Init(Tcl_Interp *interp)
{
#ifdef USE_TCL_STUBS
  if (Tcl_InitStubs(interp, "8.5", 0) == NULL) {
    return TCL_ERROR;
  }
#endif

  if (Tcl_PkgProvide(interp, "vtcl", "0.0.1") != TCL_OK)
    return TCL_ERROR;

  Tcl_CreateNamespace(interp, "vtcl", (ClientData) NULL,
                          (Tcl_NamespaceDeleteProc *) NULL);

  Tcl_CreateObjCommand(interp, "::vtcl::recordings", ListRecordingsCmd,
                       (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL);

  return TCL_OK;
}

Answer 1

The management of the Tcl_Obj * reference counts looks absolutely correct, but I do wonder whether you're freeing all the other resources associated with a particular instance in your real code. It might also be something else entirely; your code is not the only thing in Tcl that allocates memory! Furthermore, the default memory allocator in Tcl does not actually return memory to the OS, but instead holds onto it until the process ends. Figuring out what is wrong can be tricky.

You can try doing a build of Tcl with the --enable-symbols=mem passed to configure. That makes Tcl build in an extra command, memory, which allows more extensive checking of memory management behaviour (it also does things like ensure that memory is never written to after it is freed). It's not enabled by default because it has a substantial performance hit, but it could well help you track down what's going on. (The memory info subcommand is where to get started.)

You could also try adding -DPURIFY to the CFLAGS when building; it completely disables the Tcl memory allocator (so memory checking tools like — commercial — Purify and — OSS — Electric Fence can get accurate information, instead of getting very confused by Tcl's high-performance thread-aware allocator) and may allow you to figure out what is going on.

Answer 2

I found where the leak is. In function ListRecordingsCmd, I replaced line

Tcl_AppendObjToObj (cmd, Tcl_NewIntObj (obj_id++));

with

Tcl_Obj *obj = Tcl_NewIntObj (obj_id++);
Tcl_AppendObjToObj (cmd, obj);
Tcl_DecrRefCount(obj);

The memory allocated to store the object id was not released. The memory used by the tclsh process is now stable.