Is web crawling and data scraping part of your daily routine? If so, you already know that most of the websites implement firewalls that automatically block repeated requests coming from the same IP address. The solution? Regularly change your public IP address. Learn how to automatically change IP every X minutes and increase your productivity.
Why automatically change IP?
There are 2 main reasons for changing your IP automatically every several minutes: to increase the level of anonymity or to automate tasks like web crawling or data scraping.
If you are looking for increasing your online anonymity, the automatic IP rotation virtually changes your online identity and hides your public IP address. Thus, tracking your browsing or downloading activities becomes more difficult (unlikely, but not impossible).
For data scraping and web crawling the frequent change/hide IP address is the way to go. Can it be done manually? Yes. Is it efficient? No. The auto IP rotation is the good solution to the problem.
Automatically change IP prerequisites
The auto rotate IP is not a trivial task and the implementation depends on your needs. Before you proceed, you need to ask your self the followings: how many IPs you need, on what locations (countries, states) and what is the desired rotating period.
Number of IPs
As mentioned above, if you are in the data scraping business than you need enough different IPs such that you don’t get blocked. How many exactly? It depends on the site you are scraping, on the site firewall, on the scraping frequency, etc. I guess that you need between 20 to 100 IPs for simple tasks, and more for complex data requests.
Number of locations
You may need to change your virtual location with IPs from different parts of the World. Yes, it is possible, but you have to find the proper solution.
How often do you need to have your IP automatically changed? Once every minute, every 5 minutes, once an hour? Some of the solutions to be presented have certain restrictions regarding the rotating frequency, other are quite flexible. Keep reading.
Methods for rotating IP
I am going the present several solutions that are based on the following approaches:
- Proxies – free solution, requires some manual work, premium proxies available.
- Tor – free solution, depends on the quality of the Tor nodes, mode difficult to setup.
- VPN services – the most reliable and safe solution, but the most expensive.
Take a look to the table below that summarizes the main features of the proposed approaches:
|IP switch speed||Fast||Medium||Slow|
|Control over the IPs and locations||Yes||No||It depends on the provider|
|Works with||Browsers (only)||All apps||All apps|
1. Elite Proxy Switcher
The first application that I recommend uses proxies: Elite Proxy Switcher. Basically, it tests proxies and rotates them. For the free version you need to get your own proxies from proxy list sites. Take a look below to the video tutorial for more details:
After you get the working proxies you may start rotating them: go to Switch > Use automatic switch and set the switch interval (in seconds). Yes, you may change the IP every second if you need to.
That’s it! You get the auto change IP feature. Elite Proxy Switcher also offers premium proxies that are already tested daily.
The main downside of the free proxies is that they have a short life and they are most of the time unreliable. Thus, you need to check them regularly and add new ones all the time. Another issue (valid for all proxy-based solutions) is that they work only for browsers (Chrome, Firefox, Internet Explorer/Edge), not for other applications. Thus, if you use a special app that performs the data scraping the Elite Proxy Switcher will not work.
2. TOR + Privoxy
This solution is not easy to be implemented as it request some technical knowledge. If you are not familiar with Tor and you don’t have any Linux skills than this is not for you.
The main advantage of the Tor + Privoxy approach is that it is that you get the auto change IP for free. The downsides include the fact that you can not control the IPs and the locations you get (sometimes, even if you change the node you may get the same Tor IP address). Also, the Tor exists may be slow.
Next, a VPN-based solution to automatically change IP: ibVPN, a nice VPN service that may be used to rotate IP addresses.
Start with a free trial from the ibVPN site and install the All-In-One VPN client for Windows (the MacOS app does not include the rotating servers option, at least not now). After login, go to Servers, select the desired locations/servers as Favorites and click on Rotating Favorites. Eventually, click Connect. The app will change your IP every X minutes (the value of X can be setup from Settings tab from predefined values: 5, 10, 15, 30, 60 minutes). Thus, you may schedule IP address change as often as you need.
In order to make sure that the web crawler or data scraper app does not use your real IP I recommend you to switch ON the KillSwitch (Internet KillSwitch), a function that cuts the Internet connection while the IP is changed. It worth mentioning that the IP switch is slower when using VPNs than when using proxies, as it requires the setup of a VPN connection, a time consuming operation (5-30 seconds).
IPVanish, another top VPN provider, is also a solution to be considered for the periodic IP address change. The Windows and MacOS apps offer access to the Periodic IP address Change feature that you may use to rotate IPs. It comes with two drawbacks: the minimum rotating period is 45 minutes and you may not select the locations/servers to be rotated.
Important! Your VPN connection will be temporarily interrupted while our app changes your IPVanish IP address.
HideMyAss is (or it was) the preferred tool for those performing data scraping. Unfortunately, the latest versions of the HideMyAss apps (3.x) have removed the rotating IPs feature. Most probably it will be added in future releases.
The good news is that you may still install version 2 of the app. The rotating feature can be found in the IP address settings tab, on the left of the application. If you are using one device the minimum time of IP change can be 2.5 minutes, and if you are connecting from 2 devices you can set 5 minutes or more for the automatic IP change.
To sum up, the automatically change IP is possible. Firstly, you should know exactly what your needs are and, secondly, select the proper solution. My advice is to give a try to proxies and to ibVPN as it offers free trial. Enjoy!
Do you know other methods for automatically change IP address? Let us know your thoughts in the comments below.