Painting walls - brown to brown, or brown to white?

My walls are a great greige color. I like to change my accessories seasonally … curtains need to get washed it works well. It’s easier to change throw pillows and tablecloths than a wall color. And since I use the same color in every room I always have some on hand for any touch-ups that are needed. I find that going White shows every little speckle which is not something I want to deal with