Is web crawling and data scraping part of your daily routine? If so, you already know that most of the websites implement firewalls that automatically block repeated requests coming from the same IP address. The solution? Regularly change your public IP address. Learn how to change IP every X minutes and increase your productivity automatically.
Why automatically change IP?
There are two main reasons for changing your IP automatically every several minutes: to increase the level of anonymity or to automate tasks like web crawling or data scraping.
If you are looking for increasing your online anonymity, the automatic IP rotation virtually changes your online identity and hides your public IP address. Thus, tracking your browsing or downloading activities becomes more difficult (unlikely, but not impossible).
For data scraping and web crawling the frequent change/hide IP address is the way to go. Can it be done manually? Yes. Is it efficient? No. The auto IP rotation is an excellent solution to the problem.
Automatically change IP prerequisites
The auto-rotate IP is not a trivial task, and the implementation depends on your needs. Before you proceed, you need to ask your self the following: how many IPs you need, on what locations (countries, states) and what is the desired rotating period.
Number of IPs
As mentioned above, if you are in the data scraping business than you need enough different IPs such that you don’t get blocked. How many exactly? It depends on the site you are scraping, on the site firewall, on the scraping frequency, etc. I guess that you need between 20 to 100 IPs for simple tasks, and more for complex data requests.
Number of locations
You may need to change your virtual position with IPs from different parts of the World. Yes, it is possible, but you have to find the proper solution.
How often do you need to have your IP automatically changed? Once every minute, every 5 minutes, once an hour? Some of the solutions to be presented have specific restrictions regarding the rotating frequency, and others are quite flexible. Keep reading.
Methods for rotating IP – Automatic IP changer solutions
I am going to present several solutions that are based on the following approaches:
- Proxies – free solution, requires some manual work, premium proxies available.
- Tor – free solution, depends on the quality of the Tor nodes, more challenging to set up.
- VPN services – the most reliable and safe solution, but the most expensive.
Take a look at the table below that summarizes the main features of the proposed approaches:
|IP switch speed||Fast||Medium||Slow|
|Control over the IPs and locations||Yes||No||It depends on the provider|
|Works with||Browsers (only)||All apps||All apps|
1. Elite Proxy Switcher
The first application that I recommend uses proxies: Elite Proxy Switcher. It tests proxies and rotates them. For the free version, you need to get your proxies from proxy list sites. Take a look below at the video tutorial for more details:
After you get the working proxies you may start rotating them: go to Switch > Use the automatic switch and set the switching interval (in seconds). Yes, you may change the IP every second if you need to.
That’s it! You get the auto change IP feature. Elite Proxy Switcher also offers premium proxies that are already tested daily.
The main downside of the free proxies is that they have a short life and are most of the time unreliable. Thus, you need to check them regularly and add new ones all the time. Another issue (valid for all proxy-based solutions) is that they work only for browsers (Chrome, Firefox, Internet Explorer/Edge), not for other applications. Thus, if you use a particular app that performs the data scraping the Elite Proxy Switcher will not work.
2. TOR + Privoxy
This solution is not easy to be implemented as it requests some technical knowledge. If you are not familiar with Tor and you don’t have any Linux skills than this is not for you.
The main advantage of the Tor + Privoxy approach is that it is that you get the auto change IP for free. The downsides include the fact that you can not control the IPs and the locations you get (sometimes, even if you change the node you may get the same Tor IP address). Also, the Tor exists may be slow.
Next, a VPN-based solution to auto change IP: ibVPN, an excellent VPN service that may be used to rotate IP addresses.
Start with a free trial from the ibVPN site and install the All-In-One VPN client for Windows (the macOS app does not include the rotating server’s option, at least not now). After login, go to Servers, select the desired locations/servers as Favorites and click on Rotating Favorites. Eventually, click Connect. The app will change your IP every X minutes (the value of X can be set up from the Settings tab from predefined values: 5, 10, 15, 30, 60 minutes). Thus, you may schedule IP address change as often as you need.
To make sure that the web crawler or data scraper app does not use your real IP I recommend you to switch ON the KillSwitch (Internet KillSwitch), a function that cuts the Internet connection while the IP is changed. It worth mentioning that the IP switch is slower when using VPNs than when using proxies, as it requires the setup of a VPN connection, a time-consuming operation (5-30 seconds).
IPVanish, another top VPN provider, is also a solution to be considered for the periodic IP address change. The Windows and macOS apps offer access to the Periodic IP address Change feature that you may use to rotate IPs. It comes with two drawbacks: the minimum rotating period is 45 minutes, and you may not select the locations/servers to be rotated.
Important! Your VPN connection will be temporarily interrupted while our app changes your IPVanish IP address.
HideMyAss is (or it was) the preferred tool for those performing data scraping. Unfortunately, the latest versions of the HideMyAss apps (3.x) have removed the rotating IPs feature. Most probably it will be added in future releases.
The good news is that you may still install version 2 of the app. The rotating feature can be found in the IP address settings tab, on the left of the application. If you are using one device the minimum time of IP change can be 2.5 minutes, and if you are connecting from 2 machines, you can set 5 minutes or more for the automatic IP change.
To sum up, the automatically change IP is possible. Firstly, you should know what your needs are and, secondly, select the proper solution. My advice is to give a try to proxies and ibVPN as it offers a free trial. Enjoy!
Do you know other methods for automatically change the IP address? Let us know your thoughts in the comments below.