Variables in Prolog can only be assigned once. As soon as X has the value [1,2,3,4] it can never have another value. A temporary variable and append/3, like you mentioned, is the way to do it.
Having said that, you can do one trick which probably isn't recommended. If X = [1,2,3,4,Y] then you can do Y=5 and X now has the value you want. I believe this technique is called a difference list.