Assign your objects to different Layers.
By doing this you can either set them to collide with each other, or to ignore any collisions between them. So make an A layer for all your A objects, and a B layer for all your B objects. (Of course you can name it whatever you like) Go to
Edit > Project Settings > Physics
where you can edit the layer collision matrix, to enable or disable collisions between layer elements.