The simplest method:
0mmm 1mmm 0mmm - no repeating nibbles, fastest encoding/decoding, no checksum
But I'd recommend the following method:
You have 3600 = 16*15*15 possible nibble triplets (first nibble has 16 variants, second has 15, third has 15).
You can have 2 bits for checksum and 3600/4 = 900 domain-specific codes.
Encoder (decoder is the reverse of it):
C = 0..899 -- your code to be encoded
C = C * 4 -- appending a "hidden checksum"
N3 = C mod 15 -- 0..14
C = C div 15
N2 = C mod 15 -- 0..14
N1 = C div 15 -- 0..15
nibble1 = N1
nibble2 = (nibble1 + 1 + N2) mod 16
nibble3 = (nibble2 + 1 + N3) mod 16
Dividing by 15 is as simple as multiplying by 0.0001000100012 if you haven't DIV operation.
Decoder:
nibble1, nibble2, nibble3 -- your three nibbles
N1 = nibble1
N2 = (nibble2 - nibble1 - 1) mod 16
N3 = (nibble3 - nibble2 - 1) mod 16
C = (N1*15 + N2)*15 + N3
if C mod 4 != 0 then CHECKSUM ERROR
C = C/4 -- 0..899
UPD for new conditions:
no checksum, 8*14*8 = 896 possible codes
Encoder (decoder is the reverse of it):
C = 0..895 -- your code to be encoded
N1 = C mod 8
C = C div 8
N2 = (C div 8) + 1 + N1
N3 = (C mod 8) + 8
if N2 >= N3 then N2 = N2 + 1
nibble1 = N1 -- 0..7
nibble2 = N2 mod 16
nibble3 = N3 -- 8..15
Decoder:
nibble1, nibble2, nibble3 -- your three nibbles (0..7, 0..15, 8..15)
N1 = nibble1
N2 = (nibble2 - nibble1 - 1) mod 16
N3 = nibble3
if N1 + N2 >= N3 then N2 = N2 - 1
C = (N2*8 + N3 - 8)*8 + N1 -- 0..895