Personally I'd start by using grouping to make it more efficient (this part is optional of course) but your problems are occurring because it is averaging the seconds but reporting the others as time. Using the following answer you can convert between seconds and time allowing you to compare accordingly, alternatively you could convert your former columns to seconds.
SELECT
FastestSegmentTime = MIN(CAST(tblTrace.trFinish - tblTrace.trStart AS TIME(2)))
, SlowestSegmentTime = MAX(CAST(tblTrace.trFinish - tblTrace.trStart AS TIME(2)))
, AvgSegmentTime = DATEADD(ms, 1000 * AVG(DATEDIFF(SECOND, tblTrace.trStart , tblTrace.trFinish)),0)
FROM
tblTrace
INNER JOIN tblUsers ON usrID = tr_usrID
WHERE
(tblTrace.trFinish IS NOT NULL)
AND (tblTrace.trObjectType LIKE 'Segment%')
AND (tblTrace.tr_vnuID = @vnuID)
AND (tblTrace.trStart BETWEEN @StartDate AND @EndDate)
AND (tblUsers.usrEmail NOT LIKE '%@test%')