Dealing with attributes as filters in search and category level

Question

So, i m really confused with creating attributes and sets in magento. The Problem is let conside size attribute. I have two type of Product subcategories "Shirts" and "Shoes" under one "Fashion Category" . Now since size attribute shud be different for "shoes" and "shirts" i created two version of it "shirt_size" and "shoes_size". Now the Problem is with Filter in Layered Navigation. On Main Category level, there are two SIZE filters available. (provided i have labeled both attributes as SIZE for frontend). So this is confusing. I Really dont want to disable them as then they will stop showing at "shoes" or "shirt" category level too.

Same thing repeats for several other attributes like "Brand" etc (megastore project)

I have seen this thread Filter only in Last level , but i do want some of choosen filters like "Price" and other some to show up in both search and category filters.

What should be my solution approach? I m struggling to configure the store Thanks