shared memory vs distributed memory and multithread vs multiprocess

Question

I am learning parallel programming by myself. I wonder if distributed memory is always multiprocess and multithread is always shared memory? if multiprocess can be both for distributed memory and for shared memory? Thanks and regards!

Answer 1

Yes, yes, and "yes, in a sense"

In a distributed memory system, different CPU units have their own memory systems. Access from another CPU will most likely be slower or with a more limited coherency model, if indeed it is possible at all. This will be more typical of a message-passing multiprocessor.

Using multiple threads for parallel programming is more of a software paradigm than a hardware issue, but you are correct, use of the term thread essentially specifies that a single shared memory is in use, and it may or may not include actual multiple processors. It may not even include multiple kernel threads, in which case the threads will not execute in parallel.

I'm not completely clear on the meaning of last question. Certainly, by saying "distributed memory" or "shared memory" it implies "distributed over processors" and "shared by processors", so I suppose the terms are only reasonably applied to multiprocessor or potentially multiprocessor systems. If we are talking about multiple processes in the software sense, I guess it's pretty much a requirement for distributed memory systems, and essentially a requirement (they might be called threads) for a shared memory system.

I should add that distributed-memory-but-cache-coherent systems do exist and are a type of shared memory multiprocessor design called NUMA. Only a few years ago these machines were the lunatic fringe of parallel computing, but now the Intel Core i7 processors have brought NUMA into the mainstream.