I'm not sure of your height, but I know for a fact I had a similar mindset such as yourself. I'm 6'2, and I use to weigh 170 (went down from 250) with that height and still thought I was fat! For the past 8 months though I have been bulking and I must say its a much different feeling (Weigh 200 now). Fat doesnt come on as easy, more energy, not obsessive over the foods I eat (just cautious), and not to mention some nice muscle
Personally I think you should begin a bulking diet, eat big and clean, workout using compound lifts. Just PM me if you want some programs, I'l point you to a few
If you are anything like me, youll get very good results.