I have an awful time
gaining weight. All I get from a layoff is a change in the quality of my skin. I actually lose a few pounds when I lay off from training. The muscle doesn't come off for a long while though.
Any weight changes within 3-5 lbs don't concern me. It's often dehydration.
As an example my workout schedule has been more sporadic over the last month yet I dropped another pant size (to a 4 now ?!?).
If you are maintaining a pretty healthy weight/bodyfat level for your physique the fine tuning your looks and weight comes more from dieting than from exercise anyway.