I was googling this topic as my husband and I are in a similar situation. We only have two kids and found out a couple days ago that we are pregnant with our 3rd (oops!) We are both struggling with it, but know that we could never live with ourselves if we opted not to have it. I also have medical issues (a couple autoimmune diseases) which is really where our concerns lie. Both of my pregnancies were straight drama and I imagine this one will be no different. I look at my boys now and know that they misery of pregnancy was worth it, but boy will it be a houseful!
I am not entirely sure what your situation is, but things always have a way of working themselves out, I have faith. Talk with a person that is not your husband and get your feelings out there if nothing else than to hear your thoughts outside your head. It's ok to be selfish, being a parent is HARD!
Just don't let it tear apart your existing family. Bottom line- it's just a baby right?