RTOS usage when programming in assembly?

Question 1

It actually depends on the OS. Most known OSes allows using assembly language programming, but some of them makes it very uncomfortable (for example Mac OS X, that puts very strange requirements for the stack alignment on API calls).

I can imagine some OS that will make assembly programming absolutely impossible, but such OS probably will limit itself as well in some features.

So, the rule is - if the OS allows running of compiled binary files, then it allows assembly language programming.

It is completely another talk how easy is to create such programs. Good API documentation is essential.

Question 2

Certainly the answer is yes. How that is done may depend on the selected architecture, and the specific RTOS.

Most RTOS kernels are provided as static link libraries to which you link your application code to form a monolithic load image. A few such as QNX are full operating systems that dynamically load and execute applications at runtime. In the latter case, making OS calls from assembler should be dealt with in the OS documentation. In the case of a statically linked RTOS library, the assembler interface will normally conform to the ABI and calling convention of the target architecture, and this will be documented for the architecture, and possibly the RTOS itself.

Most RTOS products are designed with a C API interface, documentation for your target on calling C code from assembler then applies. You may find this information in the assembler or RTOS documentation.

With all that said, the argument for using assembler is usually to maintain tight control over code size and performance, but by using a large(ish) third-party library, you to some extent loose that control, and might arguably do as well to simply use C or C++.

The truth is that in most cases you need to be highly knowledgeable about a specific instruction set to beat an optimizing C compiler in both performance and code size, and even if you have that knowledge, to hand optimise large bodies of assembler is seldom worth the effort from a productivity point of view. In large assembly code bases, it is common for reasons of productivity to use a large amount of boiler-plate and macro generated code - this will often be sub-optimal for a specific use while a compiler optimiser can consider the implementation of every part of the code during translation. See this article on embedded.com by Colin Walls (read the comments too - including mine - for balance, and just for the fun of embedded geeks in disagreement).