I figured it out, I needed to remove all remoteviews each time I wanted to update notification:
notRemoteView = new RemoteViews(context.getPackageName(), R.layout.notification_layout);
notRemoteView.removeAllViews(R.id.image_button_container);
Thanks to this answer: Android app widget: content added twice