Shouldn't every algorithm run in pseudo-polynomial time?

Question

Wiki says a numeric algorithm runs in pseudo-polynomial time if its running time is polynomial in the numeric value of the input, but is exponential in the length of the input – the number of bits required to represent it.

This answer very well explains why primality test algorithm of running time $O(n)$ is actually pseudo-polynomial time. Simply put A very simple way of thinking about it is that if you double the limit, the size of the input only increases by one bit (since the limit is part of the input), while the run time is doubled. That is clearly exponential behavior with respect to the input size.

That is, add one extra bit in the bit representation of your input and the running time will increase exponentially.

Okaaay, My doubt is, isn't it applicable to every algorithm then? Like in sorting. One might say "The main difference is that, in pseudo-polynomial time algorithms, n represents a number in the input, while in other algorithms (sorting) n represents the number of things"

To which I think, how does that matter? Searching maximum in an unsorted array of size $n$ takes $O(n)$ time, add one extra bit to $n$ and the running time will grow exponentially. Hence Shouldn't every algorithm run in pseudo-polynomial time?