In general, "X is closed under Y" means that if you take some Xs and Y them, the result is an X. For example, "the set of integers is closed under addition" means that if you take two integers and add them, the result is an integer.
Therefore, saying that the class Applicative
is closed under composition means that if you take two applicative functors and compose them, the result is also an applicative functor.