Why does dm_os_memory_clerks have MEMORYCLERK_SQLCLR values on one NUMA node?

Question

We are using a SQLCLR stored procedure for service broker activation, and I want to monitor the memory used by the CLR code. Looking at sys.dm_os_memory_clerks, I see that only NUMA node 1 has any pages associated with the MEMORYCLERK_SQLCLR type. The server has two 8-core CPUs and is running SQL 2014 CU6.

Is this expected? Or should I see memory used on both nodes like I do with MEMORYCLERK_SQLBUFFERPOOL?

Query:

SELECT DOMC.memory_node_id
    , DOMC.pages_kb
    , DOMC.virtual_memory_reserved_kb
    , DOMC.virtual_memory_committed_kb 
FROM sys.dm_os_memory_clerks DOMC where type = 'MEMORYCLERK_SQLCLR'

Results:

memory_node_id pages_kb             virtual_memory_reserved_kb virtual_memory_committed_kb
-------------- -------------------- -------------------------- ---------------------------
0              88232                12607744                   1408652
1              0                    0                          0
64             0                    0                          0

Answer 1

Unfortunately, there is not a whole lot of information out there on the specifics of SQLCLR memory usage. I did, however, find the following two resources helpful as a starting point:

• Memory Usage in SQL CLR
• SQL Server: Clarifying The NUMA Configuration Information

From those and looking through various DMVs and the MSDN pages explaining (well, it's not much explaining, but technically it's more than nothing) how those DMVs kinda sorta work, I put together the queries shown below. From running those on a few different systems, it seems that SQLCLR memory allocation is concentrated on one particular memory_node_id, but there is still some allocation on other nodes, at least in terms of Proc Cache. So for now, unless someone has evidence or information to the contrary, what you are experiencing is "expected".

---

This query reports the base info:

SELECT domc.*
FROM   sys.dm_os_memory_clerks domc
WHERE  domc.[type] LIKE '%CLR%'
AND    domc.[memory_node_id] <> 64;

You will notice that by not specifying one particular [type], there are a few more that can take up some memory. Total 4 types that I have seen are:

• MEMORYCLERK_SQLCLR,
• MEMORYCLERK_SQLCLRASSEMBLY
• CACHESTORE_CLRPROC
• CACHESTORE_CLRUDTINFO

If you are wondering why I am filtering out [memory_node_id] of 64, run this:

SELECT * FROM sys.dm_os_nodes;

and you will see that node_id of 64 is DAC (Dedicated Administrator Connection).

The following will give you the totals for the two important fields. Please note that the fields changed in SQL Server 2012 so you need to pick the one that is appropriate for the version of SQL Server it is being used on:

-- SQL Server 2012, 2014, and 2016:
SELECT SUM(domc.pages_kb) AS [TotalPagesKb],
       SUM(domc.virtual_memory_committed_kb) AS [TotalVirtualMemoryKb]
FROM   sys.dm_os_memory_clerks domc
WHERE  domc.[type] LIKE '%CLR%'
AND    domc.[memory_node_id] <> 64;

-- SQL Server 2005, 2008, and 2008 R2:
SELECT SUM(domc.single_pages_kb + domc.multi_pages_kb) AS [TotalPagesKb],
       SUM(domc.virtual_memory_committed_kb) AS [TotalVirtualMemoryKb]
FROM   sys.dm_os_memory_clerks domc
WHERE  domc.[type] LIKE '%CLR%'
AND    domc.[memory_node_id] <> 64;

The next query shows the relationship between "memory_clerks" and "memory_objects":

SELECT domc.*, N' █ ' AS [ █ ], domo.*
FROM   sys.dm_os_memory_clerks domc
LEFT JOIN sys.dm_os_memory_objects domo
        ON domo.page_allocator_address = domc.page_allocator_address
WHERE  domc.[type] LIKE N'%CLR%'
AND    domc.[memory_node_id] <> 64;

The following queries are a more focused version of the first query (and more like the query in the Question), but the fields changed in SQL Server 2012 so you need to pick the one that is appropriate for the version of SQL Server it is being used on:

-- SQL Server 2012, 2014, and 2016:
SELECT domc.memory_node_id,
       domc.[type],
       domc.pages_kb,
       domc.virtual_memory_committed_kb
FROM   sys.dm_os_memory_clerks domc
WHERE  domc.[type] LIKE '%CLR%'
AND    domc.[memory_node_id] <> 64;

-- SQL Server 2005, 2008, and 2008 R2:
SELECT domc.memory_node_id,
       domc.[type],
       (domc.single_pages_kb + domc.multi_pages_kb) AS [pages_kb],
       domc.virtual_memory_committed_kb
FROM   sys.dm_os_memory_clerks domc
WHERE  domc.[type] LIKE '%CLR%'
AND    domc.[memory_node_id] <> 64;

---

UPDATE:

I did some additional testing on a dual-processor system by creating 2 databases and, using a static variable, loaded 80 MB into each of the databases. I created two databases instead of one to see if an AppDomain had to remain on one memory node, if it would be able to at least place another AppDomain on another memory node in order to spread things out. The results:

• All 160 MB across both DBs (and hence across both AppDomains) went onto memory node 0
• All 160 MB went into virtual memory, not physical memory. It was only the [virtual_memory_committed_kb] field in sys.dm_os_memory_clerks that increased. And the "Commit Size" column in "Task Manager" also reflected this increase.