Physical Boundaries

My husband and my son both have adhd. My son was a huge handful to raise. We worked 100x harder raising him than we did our daughter. We managed to keep him on track fairly well while he lived with us throughout grade school. After he left for college (10 hours away) he became a drug addict. Two years ago he took LSD and went into a deep state of psychosis. He stayed in psychosis for several months and spent 20 days in a mental hospital. He has relapsed several times since then and had another bout of psychosis after using edibles. As you can imagine, it's been a chaotic nightmare. 

He's now 23 years old and currently living and working 25 miles from where I live with my husband. I feel the need for some physical boundaries from the chaos that my son has the ability to create. I want to keep the code to the entry door of our house private. I can set up a temporary code from my phone and give him access at any time. I just don't want him to have unlimited access without my knowledge. My husband has shamed me and tried to make me feel guilty for this decision.

What do you think? Am I being unreasonable by wanting and needing some physical boundaries?