The kernel maintains a per-process table of file descriptors.
Otherwise a multi-user multi-process system could not even work, as the STDIN, STDOUT and STDERR file descriptors (0, 1, 2) of all processes would have to point to the same file (terminal), but they are obviously using independent terminals.
On the other hand, when you fork, the child process inherits all open file descriptors from the parent and they point to the same physical file. But files opened after the fork are independent.