The value you passed to *open()
(call it mode
) is modified in this way:
perms = mode & ~umask;
Meaning every bit in umask
that is set turns off the corresponding bit in the resulting permission, regardless of whether it was set or not. In your particular example:
0644 = 0b110100100
07 = 0b000000111
~07 = 0b111111000
0644 & ~07 = 0b110100000 = 0640