My computer has only 4 GB of RAM and when I build (amd64) I usually have firefox and google chrome open at the same time, plus I am usually either straming music with Amarok or watching some MOOC videos.
My swap memory is hardly used, just occasionaly.

So perhaps there is something else going on, Michael.

cheers
  Michele