Haven't found a way to format character strings using the LabelFormat
property.
One option is to use an expression in the Label
property of the category group to limit the number of characters displayed. The below example limits the label string to 5 characters:
Expression
=LEFT(Fields!CategoryGroupField.Value, 5)