Yes, it can be done. Not to the point of deleting your key (that makes no sense - you need the key), but ssh-agent is what you want. Add it to your shell config and it will only ask to be unlocked once per however often you define.
I have this function defined and called:
<span style="color:#323232;">function ssh-agent-setup() {
</span><span style="color:#323232;"> # SSH agent
</span><span style="color:#323232;"> pid_file="$HOME/.ssh/ssh-agent.pid"
</span><span style="color:#323232;"> SSH_AUTH_SOCK="$HOME/.ssh/ssh-agent.sock"
</span><span style="color:#323232;"> if [ -z "$SSH_AGENT_PID" ]
</span><span style="color:#323232;"> then
</span><span style="color:#323232;"> # no PID exported, try to get it from pidfile
</span><span style="color:#323232;"> SSH_AGENT_PID=$(cat "$pid_file")
</span><span style="color:#323232;"> fi
</span><span style="color:#323232;">
</span><span style="color:#323232;"> if ! kill -0 "$SSH_AGENT_PID" &> /dev/null
</span><span style="color:#323232;"> then
</span><span style="color:#323232;"> # the agent is not running, start it
</span><span style="color:#323232;"> rm "$SSH_AUTH_SOCK" &> /dev/null
</span><span style="color:#323232;"> >&2 echo "Starting SSH agent, since it's not running; this can take a moment"
</span><span style="color:#323232;"> eval "$(ssh-agent -s -a "$SSH_AUTH_SOCK")"
</span><span style="color:#323232;"> echo "$SSH_AGENT_PID" > "$pid_file"
</span><span style="color:#323232;">
</span><span style="color:#323232;"> >&2 echo "Started ssh-agent with '$SSH_AUTH_SOCK'"
</span><span style="color:#323232;"> fi
</span><span style="color:#323232;"> export SSH_AGENT_PID
</span><span style="color:#323232;"> export SSH_AUTH_SOCK
</span><span style="color:#323232;">}
</span><span style="color:#323232;">
</span><span style="color:#323232;">ssh-agent-setup
</span>
This way it stores the unlocked key in memory until the end of the session.