If your network implements DiffServ, then you can use setsockopt() with IP_TOS to set the DiffServ code point (DSCP) on your packets to a suitable value. (Before DiffServ was specified, these bits were interpreted as Type of Service (TOS) and priority fields, hence the name.)
Note that the UDP-based transport protocol (uTP) in modern BitTorrent implementations already behaves as if it had lower priority than "normal" traffic, and will back off when it senses congestion.