java.lang.ClassCastException: java.util.Collections$UnmodifiableMap cannot be cast to Param

Question 1

The problem is here in the var attribute.

<f:selectItems value="#{mainBean.params}"
    var="param" 
    itemValue="#{param}" 
    itemLabel="#{param.name}" 
/>

The #{param} is a reserved EL variable referring the ServletRequest#getParameterMap() which is of type Map<String, String[]> and indeed an unmodifiable map.

You shouldn't be using reserved EL variable names for own variables. Rename it to something else, e.g. _param or so.

<f:selectItems value="#{mainBean.params}"
    var="_param" 
    itemValue="#{_param}" 
    itemLabel="#{_param.name}" 
/>

Your converter, albeit a bit clumsy and not entirely robust, is totally fine.

Question 2

Here check that working code and try to find a mistake yourself. It will make you remember it harded:)

                                <h:selectOneMenu rendered="#{categoryConverterBean.rendered}" class="subcategory" id="subcategory" converter="#{subcategoryConverterBean}" value="#{auctionBean.subCategory}">  

                                    <f:selectItems 
                                        value="#{categoryListBean.subcategory}" 
                                        var="subcategory"
                                        itemLabel="#{subcategory.subName}" 
                                        itemValue="#{subcategory}" 
                                        />
                                </h:selectOneMenu>

auctionBean (nothing special):

public Subcategory getSubCategory() {
    return subCategory;
}

public void setSubCategory(Subcategory subCategory) {
    this.subCategory = subCategory;
}

subcategoryConverter:

@ManagedBean(name = "subcategoryConverterBean") 
@FacesConverter(value = "subcategoryConverter")
public class SubcategoryConverter implements Converter {

    @PersistenceContext()
    private transient EntityManager em;

    @Override
    public Object getAsObject(FacesContext arg0, UIComponent arg1, String value) {
        // TODO Auto-generated method stub
            System.out.println("wybrales: " + value);
        Query query = em.createQuery("SELECT c FROM Subcategory c WHERE c.subId=?1");
        query.setParameter("1", Integer.parseInt(value));

        return query.getSingleResult();
    }

    @Override
    public String getAsString(FacesContext arg0, UIComponent arg1, Object value) {
        // TODO Auto-generated method stub
        return ((String) String.valueOf(((Subcategory) value).getSubId()));
    }

}

and finally categoryListBean:

@ManagedBean
public class CategoryListBean {

    private List<CategoryBean> category;
    private List<SubcategoryBean> subcategory;

    @EJB
    CategoryEAO service;

    public List<CategoryBean> getCategory() {
        category = service.getCategories();
        return category;
    }

    public void setCategory(List<CategoryBean> category) {
        this.category = category;
    }

    public List<SubcategoryBean> getSubcategory() {
        subcategory = service.getSubcategories();
        return subcategory;
    }

    public void setSubcategory(List<SubcategoryBean> subcategory) {
        this.subcategory = subcategory;
    }
}

I would focus on your getAsObject method because it looks like you dont get Param object as you think you do