I don't normally freak out about weird medical issues, but you never know with lumps, do you? I have a lump on the middle of my chest. It is small, flat and it is under the skin. You can't see it, but it's easy to feel, and it's moveable underneath the skin. I already looked the possibilities up on the internet and I am guessing it is a lipoma. It can't be cancer unless it is attached to the skin, right?
I principle I think you are right, but if you are at all worried then I guess a visit to the doctor will do you no harm, you know what I mean? I mean, it will not cost you anything if you are insured, and you will know you can stop worrying... or that you do need to get treatment. I say go to the doctor.