Few ways of going about this. One way is to have a single object that has "isRemoved" boolean on it that you just turn on and off. Another way is you could have 2 observablecollections one holding the added, and one holding the removed. so for example:
class:
public class MyData
{
public bool isRemoved { get; set; }
public string Name { get; set; }
}
Use:
ObservableCollection<MyData> AllData = new ObservableCollection<MyData>()
AllData.Add(new MyData(){ isRemoved = true, Name = "Data1"}
AllData.Add(new MyData(){ isRemoved = true, Name = "Data2"}
AllData.Add(new MyData(){ isRemoved = false, Name = "Data3"}
AddingList.ItemsSource = AllData.Where(srch => srch.isRemoved == false);
RemoveList.ItemsSource = AllData.Where(srch => srch.isRemoved == true);
In your Remove button click you just set the isRemoved to true and in your Add you set the isRemoved to false.
or you could use 2 ObservableCollections adding and removing from each.