If you require the ability to programmatically access the full power of the web browser without ever opening a web browser you should take a look at PhantomJS http://phantomjs.org/. Want even more power ? Combine it with CasperJS http://casperjs.org/.

With these technologies you can run automated scripts that don’t require any desktop UI or browser and that (as far as the remote HTTP server is concerned) are the same as opening a web browser and doing whatever tasks you would have manually done on it. You can programmatically invoke JavaScript within the context of the remote page, download files, even take screen shots of the page that is being viewed and to think that all this can be done as some background script on a GUIless virtual machine running somewhere on the cloud – isn’t technology something ? 🙂

quick installation on Ubuntu ∞

This is for version 1.9.2 – use the one that’s relevant for you:

cd ~
wget https://phantomjs.googlecode.com/files/phantomjs-1.9.2-linux-i686.tar.bz2
tar jxvf phantomjs-1.9.2-linux-i686.tar.bz2
ln -s `pwd`/phantomjs-1.9.2-linux-i686/bin/phantomjs /usr/local/bin/phantomjs

git clone git://github.com/n1k0/casperjs.git
ln -sf `pwd`/casperjs/bin/casperjs /usr/local/bin/casperjs