Withholding sex/no longer attracted to me

My husband has been withholding sex for several years because he says he no longer is attracted to me due to my forgetfulness and not doing things the way he wants me to. I have forgotten things in his lunch and today I mailed a letter but neglected to send it three day mail like he wanted me to. We've been married for twenty years and he keeps saying he should just leave because he can't trust me to do anything right, that he might as well be single if he has to do everything himself. I don't know why I stay. I feel like such a failure in life. There will be time I go with long stretches without messing anything up but then something will happen and I'm back to square one. It's so demoralizing. He'll say I'll be like My Mom and my sister who both got divorced. I honestly don't know how to go out on my own. I'm 53 years old. We don't have any kids. Most everything is in his name. How can I afford an apartment? I've only lived alone for a couple of months before we were married.