First of all note that MPLAB C18 has some divergences to the C standard. Read it's user manual, especially section 2.7 labeled "ISO divergences".
So pay attention to that C18 does char promotion instead of integer promotion which also applies to literals. So if you add 0x80 and 0x80 together you get 0x00. The way to avoid this is the same as how it can be worked around normally if you need larger constants Use 'L' or 'UL' postfixes to force calculating in 32 bits, maybe also casts where needed.
The compiler's optimizer may be somewhat imperfect, you will need to check the disassembly. If you work with 32bit literals, it may emit 32bit code even if, say, only 8bit would be needed. You may work this around by casting the result of a calculation such as:
#define C8BIT ((unsigned char)(((0x12UL * 0x100UL) + 0xC0UL)>>6))
(Just an example arbitrary calculation which would overflow otherwise)
In the example you posted the problem is similar, you essentially work with unsigned chars, the compiler promotes to unsigned char, and you except it to be calculated with at least 16bit depth. A proper and clean solution for the problematic line might be:
red = (0x100UL * (unsigned long)(t[0])) + ((unsigned long)(t[1]));
If performance matters however (assuming t[0] and t[1] there just got some example values, so may get arbitrary contents in the real program) you may want to check disassembly and finally resort to this:
red = (0x100U * t[0]) + t[1];
or
red = (((unsigned short)(t[0]))<<8) + t[1];
A sane compiler (even including C18) should give proper result on these. I also showed a version with shifting as C18 is not too strong on optimization and might include a library call to a multiply routine there. I omitted casting t[0] and t[1] as the compiler should properly promote them (the 0x100U demands at least 16bit promotion), and the cast might confuse the compiler to add extra 16bit logic (C18 is very weak in this term!) with literal '0' operands. If performance matters, check disassembly, otherwise just go with the clean solution!
You may also want to check that BYTE type. It should be unsigned char, but not necessarily. You might want to define types with fixed sizes (such as typedef unsigned char uint8;
and so) for a consistent use.