How to stop Gnome from loading
sudo apt-get install sysv-rc-conf sudo sysv-rc-conf
Use the arrows to go down to “gdm” and unselect it for runlevel (column) 2 (the others are optional, you wont boot to them unless specifically specified).
GDM (the login manager) wont start on startup no more now, and thus neither will X.
To start it occasionally:
sudo /etc/init.d/gdm start
If it’s upstart then remove gdm.conf from /etc/init
Or maybe lighted.conf in 11.10