In addition to your syntax errors that @Gordon already pointed out, it is regularly a good idea to exclude empty updates:
UPDATE transaction t
SET quantity = sub.max_quantity
FROM (
SELECT item_name, max(quantity) AS max_quantity
FROM transaction
GROUP BY 1
) sub
WHERE t.item_name = sub.item_name
AND t.quantity IS DISTINCT FROM sub.max_quantity;
No need to write new row versions (at almost full cost) without changing anything. (Except if you want to fire a trigger.)