This question is best asked with another question: what is an Operating System? Or if you prefer: where do you print the line between OS and standard libraries?
Technically, malloc
is part of the standard C library. And since the Linux is mainly written in C, and that the same library also includes many system calls, not in the C language, it is reasonable to think that this library is part of the OS.
But, on the other hand, there are several implementations of the C library, and also, the GNU C library is available for others operating systems, such as Windows. And I'm sure that there are other languages out there that call the OS without using the standard C library. So, from that POV, it is not part of the OS.
But then, Linux is the kernel, the OS should be named GNU/Linux (citation needed). But again, there are Linux systems without GNU, such as Android...
The conclusion is: the term "Operating System" is not a technical one. If you want to be precise, use kernel or standard C library, etc.