using lorazepam for a long time

I have started using Lorazepam about 3 years ago because I couldn't sleep and also had severe anxiety problems at college. Since then I have rarely went to sleep without it. I usually take 0.5 mg, but sometimes it takes a whole pill (1 mg) to put me to sleep (very rarely). However, I have recently noticed changes. I have begun to forget things, usual things, not drastically, but still concerning, and also change in a approach to things, being generally not able to get excited (or upset) about anything. I have also noticed that my potential for higher abstract thinking, like math has decreased considerably. I know that Lorazepam causes short term decrease in intellectual power, but a friend of mine has told me that used this long will cause permanent damage. I am 23 now but am afraid that if this is true my future work and capacity to achieve will be seriously hindered.

Please, help me.