Byte addressable vs Word addressable

Question

I am trying to understand the difference between byte addressing and word addressing.

A 4-way set-associative cache memory unit with a capacity of 16 KB is built using a block size of 8 words. The word length is 32 bits. The size of the physical address space is 4 GB.

No of sets in the cache $= (16 * 1024) / (4 * 8 * 4) = 2^7$

If word addressing is used :

Block offset $= 3 \ bits$

Since PAS is $4 \ GB$, total no of addresses = $2^{32} / 2^2 = 2^{30}$

So, total address bits $= 30\ bits$

Address structure :

Tag bits : $20 \ bits$
Set bits : $7 \ bits$
Block offset bits : $3 \ bits$

Now, suppose the CPU wants to access the 3rd byte of a particular word.

1. The cache controller will use the $7 \ bits$ set-field to index into a set and then comparing the higher $20\ bits$ tag-field with all of the $4$ blocks in the set. If a match is found, then cache hit occurs and the lower $3 \ bits$ block offset to put one of the word out of the $8$ words in one of the general purpose register. The CPU then extracts the 3rd byte from the word and perform the operation.
2. If tags are unmatched, then cache miss occurs, a memory read signal is sent and due to spatial locality of reference, a block containing the word is transferred into the cache.

If the CPU is byte addressable:

Total address bits $=32$

Address Structure : Tag bits : $20 \ bits$
Set bits : $7 \ bits$
Block offset bits : $5 \ bits$

If the CPU wants to access 3rd byte of a word:

1. Same as in Step 1 of word addressable, but the CPU now can directly address the 3rd byte of the word, using the lower $2 \ bits$ byte offset. However, I'm confused how that would happen. Since, the CPU register size has a width of 1 word, similar to the word addressing, one word out of the 8 words in the block will be transferred to the register. But how would the "byte extracting" step be easier here? And why do we call it byte addressing if still we are actually addressing a word?
2. Same as in Step 2 of word-addressing. Block of data will be transferred from the memory to the cache in case of cache miss.

Also, this answer says that physical memory is always byte addressable. Now, what is the difference between the addressablity of the memory and addressablity of the CPU architecture?