80legs

80legs.com provides a service for web crawling.

We put over 50,000 computers to work for you to deliver exceptional crawling performance at incredibly low costs. Our service is easy to use and completely customizable, so you can crawl and process web content however you want, whenever you want.

They announce a possibility to process “up to 2 billion web pages per day“. So this opportunity is pretty cool if you want to get and process large amount of data from dozens of web sites. But it will not work well in case when you need to scrape data just from one site. Moreover in this case you could establish a really DDOS attack for that lonely web site.  Not very good, isn’t it?

Another question is: what is the nature of their machine network? It maybe not very easy to establish such a big network at once. Even if they could the next problem is to create that network to be spreaded geographically. Because if not they should have a really high bandwidth channels. Maybe they use a zombie machine network? Anyway, it’s just my supposition and nothing else.

One more positive challenge they can propose you is an anonymity for you crawlers. Really if some web site takes care about forbidden of scrapping its own pages 80legs network could broke this at once. Because if each machine in their network has its own IP address it would be look like just dozens of common human-like users! Thus if you have some programming skills and some money you can establish highly secured and anonymous access to the WWW.

Tor project

Tor project is yet another solution to get anonymity in the WWW.  Here what they say:

Tor is free software and an open network that helps you defend against a form of network surveillance that threatens personal freedom and privacy, confidential business activities and relationships.

Tor protects you by bouncing your communications around a distributed network of relays run by volunteers all around the world: it prevents somebody watching your Internet connection from learning what sites you visit, and it prevents the sites you visit from learning your physical location. Tor works with many of your existing applications, including web browsers, instant messaging clients, remote login, and other applications based on the TCP protocol.

I have never been used Tor for myself. But idea as I get it is very good and approach of its realization is interesting. You can read more about it on Tor’s overview page. In few words it works like this. There is a set of computers that have installed Tor and all of them are united in one network. Instead of establish direct connection to some web site you can use that network. Thus you request and site’s respond go through the network of several computers. Each time you request another web page the overall path is changing. So it looks like the fluid proxy network.

Tor work process

SwissVPN

SwissVPN is a service that provides VPN access to the World Wide Web. This is quite easy and comfortable way to get to the Internet anonymously. And on the other hand is quite cheap (just 5$ per month). Every time you’re establishing connection to that service it gives you distinct IP address.

I used it for several crawler projects. If for instance web site uses captcha after intensice usage to ensure that a user is a human one, you can allow your robot to reconnect to that site through another VPN connection and get another IP address.