Before I was using eAccelerator as PHP accelerator, but then I read some reviews that XCache is the fastest, so I switched to that. For a few days everything seemed to work ok, but then noticed my VPS was slowing down. I rebooted the thing and everything was ok again. I should have realized the problem was the accelerator right from the start, but somehow I forgot I had installed it.
Anyways, finally it occured to to try to change the accelerator. I read APC will be embedded in the next PHP releases so I decided to try that one. Here’s memory usage graph of the VPS for the 7 days:

Guess at what point I switched from XCache to APC? Yep, the time when graphs went flat.
Conclusion: XCache sucks and APC rules.
Disclaimer: I’m not expert in XCache configuration so I just went with defaults. It is very possible that with right tuning XCache could have performed much better. But neither I’m expert in APC configuration, and that worked straight out of the box with default settings, which is always nice.