What loss function should I use if I have been working on a classification problem which involves both multi-label and multi-class labels?

Question

For example I have apple and pear pictures. What I am trying to do is to predict if a picture is an apple or pear picture and AT THE SAME TIME predicting whether the fruit is big and/or yellow.

Thus predicting the fruit type (apple/pear) is multi-class problem, one vs all. Predicting whether the fruit big and/or yellow is multi-label problem.

If label ordering is [apple, pear, big, yellow] then a “big yellow apple” picture should have the label [1,0,1,1]. First two part is mutually exclusive, one-hot, however last two are not.

So, what loss function should I use for this problem?