SQL Union with COUNT() Function not returning expected results

Question

I am trying to return all MSFT Lync Users who have used voice conferencing and the number of times they used it. I have to use the UNION because the table has User1 and User2 who both participated in the conference which I want in one column.

Here is the SQL:

SELECT U.UserUri,
       COUNT(U.UserUri) AS COUNT
FROM   SessionDetails AS S
       INNER JOIN Users U
         ON S.User1Id = U.UserId
WHERE  MediaTypes = 48
GROUP  BY U.UserUri
UNION
SELECT U.UserUri,
       COUNT(U.UserUri) AS COUNT
FROM   SessionDetails AS S
       INNER JOIN Users U
         ON S.User2Id = U.UserId
WHERE  MediaTypes = 48
GROUP  BY U.UserUri  

The returned results have anyone who participated in more than one conference appearing in the list twice, with two count totals. Actually, after typing this out it must be grouping by User1Id and User2ID. If a person had 9 conversations but iniated only 6 of them they would be User1 6 times and User2 3 times, creating the two different groupings.

Now I know why, but any ideas how to return no duplicate UserUri? Any help greatly appreciated!