It’s like giving a map of your infrastructure to a hacker, but it depends on your thread model. Most of the attacks on home servers are automated, so it shouldn’t be a consern
Another thing if your thread model is different, then the situation is not that good, but you can encrypt a lot of stuff, especially when you’re making your config reproducible