I need help setting boundries

I'm not very good at it.  I can set boundaries for myself but he always manages to talk me out of them with his anger.  He gets angry with me when I say No.  He feels like I'm controlling him when all I'm doing is protecting myself.  He makes me feel incredibly guilty.

Also, he rushes my "letting go" process.  We've been through a rough week this week and it still isn't over.  I still have to deal with the aftermath of his mistakes and that part has yet to come.  Once I get a letter in the mail from the apartment company telling me they received my returned check, I'm sure I will start to feel terrible all over again.  It's a trigger that will set me off.  To that end, I'm not ready to be "all in" with him again.  I'm not ready to hang out and chill like nothing is happening, but that is what he wants from me.  In the midst of a crisis he is super attentive until we have sex.  To him I think having sex seals the deal that the crisis is over and that I "love" him again.  He can let down his guard and go back to ignoring me once I've given him the green light - that being sex.

He initiated with me last night and I turned him down.  He asked me why and I told him because I am stressed out regarding our finances and I'm not in the mood.  He got really quiet.  Then I also explained to him that my fear is that he will stop paying me attention once I make love to him.  Of course he denied this, but past is prologue. 

So what do I do?  We have been getting along in spite of being in the middle of a mess, but that is partly due to the fact that it is not in my character to stay angry.  I simply don't like the feeling of anger.  However, I don't feel like everything is okay and I don't want him to feel that way either.  I need him to work on our relationship and what motivation does he have to do that if he gets all the benefits of a good relationship without working on it ie a wife with a good attitude and sex.  I do not want to use sex as a punishment, but I also do not want to make love to a man who puts very little effort into making our relationship a good one. 

I need advice.