Ok I got the anwer by pointing the Viewgroup to a specific layout
View view = inflater.inflate(R.layout.fragment_template_list,
container, false);
ViewGroup viewGroup = (ViewGroup) view.findViewById(R.id.ptr_layout);
and insert pull to refresh to that viewgroup
mPullToRefresh = new PullToRefreshLayout(viewGroup.getContext());
ActionBarPullToRefresh
.from(getSherlockActivity())
.insertLayoutInto(viewGroup)
.theseChildrenArePullable(view.findViewById(R.id.lvNameList))
.listener(this)
.setup(mPullToRefresh);