SQL Server Agent appears to be treating sysprocesses (and dm_exec_sessions) differently

Question

I have a watchdog process that runs via a SQL Agent job to manage some custom tasks. One of the checks is that the process no longer exists:

UPDATE R
SET IsOrphaned = 1, DebugTimestampUtc = GETUTCDATE()
FROM
    RequestTable AS R
    LEFT JOIN sys.sysprocesses AS P ON
        P.spid = R.SPID
WHERE P.spid IS NULL

In SSMS, I run this statement then close the query window before it completes (cancelling the query):

INSERT INTO RequestTable(SPID, CreateTimestampUtc, ...)
SELECT @@SPID, GETUTCDATE(), ...
WAITFOR DELAY '00:00:30.000'

When I manually run the query to flag orphans in SSMS it matches on the row, but my job does not, as is evidenced by the RequestTable not getting the row marked as orphaned. The job history shows it running as scheduled without any errors, taking negligible runtime (< 1 second).

Furthermore, when I augment the orphan detection statement to also find records where P.status = 'sleeping' it immediately detects when I cancel the INSERT ... WAITFOR ... query (and keep the query window open).

Is there some subtle behavior of SQL Server causing this apparently-inconsistent state of the sys.sysprocesses table between SSMS and SQL Agent? I have also tried with and without a NOLOCK query hint on RequestTable.

I'm running on SQL Server 2014 in compatibility mode for 2008R2.

(Note that using sys.dm_exec_sessions instead still behaves the same)

Thanks.

P.S. The purpose of this is to serialize asynchronous processes that have concurrency issues. Perhaps there is also a better manner to do this (without being able to fix the asynchronous processes)?

Answer 1

As pointed out in the comments by Aaron, the current session was being caught, because they are reused in the connection pool.

The solution is to filter it out:

AND session_id <> @@SPID