Would it be so bad if I took away his debit card and just gave him an allowance each month?

I am a SAHM so I do the most of the "spending" in our household.  What I mean by that is that I buy the groceries, school supplies, kids clothes, etc.  Anything we need, I make it happen, which means I do the spending. 

DH (he is the ADHD spouse) agreed to take over our banking/budgeting a while back, but has done NOTHING with it to keep it up to date.  Additionally, one of his problems is that we agree on a budget amount for "fun money" and then he spends above and beyond his allotment.  Dave Ramsey has coined the term "financial infidelity" and that's exactly what this feels like to me.

I would like to take back the budget and his debit card and instead just give him a cash allowance for two things every month: his "fun money" and gas.  Those are the only two things he needs every month.  I take care of all the other household shopping.  I would also like to get back to the cash envelope system so we can knock out our debt (again), so if he was going to get a haircut or needed some clothes, he could just grab the appropriate envelope and be on his way.  He doesn't agree with using the cash envelopes because I misplaced an envelope containing ~$100 a few months back (it was later found), but the debit card use is causing huge overspending.  We are Dave Ramsey followers and we really need to get back on a budget.  I find that the cash only method is the most effective way for me. 

He says he feels poor, but he's doing nothing to change the situation, even though he has the knowledge (we've taken FPU) and the tools.  His lack of attention to our household budget is case in point.

Would this be too controlling?  I could just fill a Visa gift card for him each month or give him cash or both.  But we need to reign this spending in if we are going to make any financial progress.  Otherwise he's just going to keep feeling poor.