You are using Barriers which cannot be used to synchronise across multiple compute cores (work groups).
Barrier synchronisation only works within the same logical work group. See this post on Khronos to get a better idea of what I am referring to.
Increasing your cluster size also increases the number of work items in use, which probably then makes use of more than one work group which is why you are experiencing this issue.
EDIT: Its probably worth pointing out that no synchronisation primitives can be used across workgroups.