As I mentioned on the phone, SSD would be the first thing I did, but RAM could be depending on the remote apps you are using. I suggested the RAM because it sounded like you were doing a lot of remoting and the video refresh rate (the way I understood what you were asking) would be mostly residing in RAM. When an application is redrawing it is using memory.
Did you ever check your swap space via the command line using the top command?
Here is a quick walkthrough: http://www.thexlab.com/faqs/lackofram.html
If you were doing more local stuff, I would say SSD first. SSD will help for sure though.