Set up a monastery in the surrounding building whose religion is centered on only letting in the one person. The 'key' is a complex ritual (like happens nightly at the tower of london) with lots of phrases and movements to learn. The ceremony happens somewhere dangerous to the person requesting access.