This should be a bit faster: (removed the inline group by, and using exists in this case can help.)
SELECT
ee_expert.expert_id ,
AVG( ee_expert_rating.rating_stars ) AS total_rating,
COUNT( DISTINCT ee_expert_rating.rating_id ) AS rating_count
FROM
ee_expert_rating RIGHT JOIN
ee_expert ON ee_expert.expert_id = ee_expert_rating.expert_id
WHERE
expert_verified_email =2 AND
expert_brief_description != '' AND
expert_account_status =1 AND
exists(
SELECT
expert_id
FROM
ee_expert_categories
WHERE
ee_expert_categories.category_id =5 and
ee_expert_categories.expert_id=ee_expert.expert_id
)
GROUP BY
ee_expert.expert_id
ORDER BY
rating_count DESC
(Try it with keeping the IN also/without the internal group by.)