毎回忘れるのでメモ代わりに。ssh-keygen で変更できます
$ ssh-keygen -p -f ~/.ssh/id_rsa
Enter old passphrase:
Key has comment '.ssh/id_rsa'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.