Subqueries are not generally allowed in aggregate functions. Instead, move the aggregate inside the subquery. In this case, you'll need an extra level of subquery because of the top 5
:
SELECT c.CategoryName,
(select sum(val)
from (SELECT TOP 5 od2.UnitPrice*od2.Quantity as val
FROM [Order Details] od2, Products p2
WHERE od2.ProductID = p2.ProductID
AND c.CategoryID = p2.CategoryID
ORDER BY 1 DESC
) t
)
FROM [Order Details] od, Products p, Categories c, Orders o
WHERE od.ProductID = p. ProductID
AND p.CategoryID = c.CategoryID
AND od.OrderID = o.OrderID
AND YEAR(o.OrderDate) = 1997
GROUP BY c.CategoryName, c.CategoryId