Python mechanize For Browsing

How to install Mechanize

we can install mechanize in two ways 

Using pip :

pip install mechanize

Or download the mechanize distribution,open it and run it:

python setup.py install

 Browsing with Mechanize 

Here is an example on how to browse webpage in python program

 import mechanize br = mechanize.Browser() br.open("http://www.example.com/")

Follow second link with element text matching regular expression

response1 = br.follow_link(text_regex=r"cheese\s*shop",nr=1)assert br.viewing_html() print br.title() print response1.geturl() print response1.info() # headers print response1.read() # body

To get the response code from a website, you can the response.code

 from mechanize import Browser browser = Browser() response = browser.open('http://www.google.com') print response.code

Get all forms from a website

 import mechanize br = mechanize.Browser()br.open("http://www.google.com/") for f in br.forms():     print f

Ignore robots.txt

br.set_handle_robots( False )

Google demands a user-agent that isn't a robot

br.addheaders = [('User-agent', 'Firefox')]

Retrieve the Google home page, saving the response

br.open( "http://google.com" )

Select the search box and search for 'foo'

 br.select_form( 'f' ) br.form[ 'q' ] = 'foo'

Get the search results

br.submit()

Find the link to foofighters.com; why did we run a search?

 resp = None
 for link in br.links():                siteMatch = re.compile( 'www.foofighters.com' ).search( link.url )          if siteMatch:                  resp = br.follow_link( link )                  break

Print the site

 content = resp.get_data() print content