As far as I understand it, the passphrase is used to encrypt the private key. ssh-agent doesn't remember the passphrase - it remembers the decrypted private key.
And, as a damienfrancois mentioned, it shouldn't remember it past a reboot.
If you wished to extract the decrypted private keys from ssh-agent
itself, you would have to find a tool written to search the memory of the running process and locate keys. One such tool can be found here, but you may well find it very challenging to use.
For a more practical answer, you can just delete your keys from ~/.ssh/id*
, make new ones that you know the passphrase for, and move on - for a new machine, you probably haven't gotten too reliant on them yet.