Bartosz Ciechanowski has your answer in incredible detail at his blog post "Exposing NSMutableArray"
tl;dr: It's a "circular buffer:"
Data Structure
As you might have guessed, __NSArrayM makes use of circular buffer. This data structure is extremely simple, but a little bit more sophisticated than regular array/buffer. The contents of circular buffer can wrap around when either end is reached.
Circular buffer has some very cool properties. Notably, unless the buffer is full, insertion/deletion from either end doesn’t require any memory to be moved. Let’s analyze how the class utilizes circular buffer to be superior in its behavior in comparison to C array.
Historically, __NSArrayM is a newer-than-Cocoa class name-- it wasn't called that in OSX 10.0, but I can't find a good link for the older names. Maybe it used to be a tree. The link in your first comment makes it sound like in 2005, at large sizes it was not a circular buffer internally.