One way of getting a very localised danger like this would be to put it near radioactive ore deposits? You can only go in with a lead shield, and have to angle it correctly
Set up a monastery in the surrounding building whose religion is centered on only letting in the one person. The 'key' is a complex ritual (like happens nightly at the tower of london) with lots of phrases and movements to learn. The ceremony happens somewhere dangerous to the person requesting access.