Try adding the fragment you want to go back to manually to the backstack.
FragmentTransaction transaction = getFragmentManager().beginTransaction();
YourFragmentName myFragment = new YourFragmentName();
transaction.replace(R.id.fragment_container, myFragment);
//if you with to add it to backStack, do this, otherwise skip the line below
transaction.addToBackStack(null);
transaction.commit();
Hope this helps.