Ultimate Guide to Leak Testing Tools
The leak test tools in this article are a suite of python tools that have been designed for both manual and automated leak testing of VPN apps. LimeVPN also has a DNS leak testing tool to help you know if your privacy is compromised. The tools were made to serve two primary purposes;
1 . To provide robust and automated deterioration testing of LimeVPN apps
2. And to facilitate leak testing practices to fix the weaknesses of the apps
Table of Content
1 . Overview of The Leak Test Tool
2. Setting Up the Leak Test Tool
3. Leak Case Study for Test Tools
The leak testing tool isn’t only beneficial to LimeVPN but to the entire VPN industry and is available open-source under the MIT license. This way, VPN users can be sure of continuous and uncompromised security and privacy as they go about their online activities. It does this by;
1 . Giving VPN users and third parties complete access to verify the quality of VPNs and access the state of their security and privacy.
2. Creating a higher yardstick for leak protection in the entire VPN industry.
3. Bring together different forms of expertise from the entire VPN industry in leak test tools.
The leak test tools can be found on GitHub. Follow the instructions in the README.md file to get started. To follow the testing instructions in the privacy research lab case studies, the leak tools would need to be installed according to the GitHub documentation. Whenever you run any of the case studies, the tools root directory refers to the location where you have cloned the git repository.
Overview of The Leak Test Tool
1 . Python
The leak test tools are written primarily in python with support tools in other programming languages when necessary. Python was used as the primary language because;
1 . The language is cross-platform and supported out of the box in most instances
2. Almost every developer knows the language and so it can be accessed readily
3. Developing the tools in this language is fast
4. It’s an expressive language and allows writing of code which is easily readable
2. Its Likened to Unit Testing
The tools were developed to run like a unit testing suite and are launched by a single script which then runs a series of tests. The tests are for different leaks under different conditions like network conditions. It was also made to be familiar with developers that work constantly with unit testing suites. So the pattern each test follows will be the familiar setup(), and teardown().
The test cases are not specific to a particular VPN configuration but were made to be generic and fit with different network configurations. When the tool is launched, it is given a command that specifies the pattern for that round of testing, and a configuration file is then given to the test suite which lists the to be run and their configurations.
The following example shows how to run the test by executing the following command from the tools root directory:
./run_tests.sh –c
Configs/case_studies/vanilla_leaks.py
Interesting Read : How to Watch Blocked YouTube Videos?
3. Device and Network Configurations
The test tool was made to run on different device configurations and the network is and since some aspects of VPN behavior that would require testing needs more than one device for the test to be done, it was designed to enable this. The device from which the tests are run is the test orchestrator, while others are simple test devices.
Simple tests are run using a single device and leaks are tested directly from the device. You can run a variety of windows tests straight from a windows device for example, and in this scenario, the test orchestrator is still the test device. Multi-device tests usually involve the test device running tests on the VPN, packet capture device, and the router.
4. Automation
The test suite was built to be automated, but would still require some manual interaction. There would be a prompt when manual interaction is needed and it will guide the user on what to do. One of the most difficult parts tests to be automated is control over VPN apps and in many tests this aspect is manually done.
Setting Up the Leak Test Tool
Detailed instructions on the set up of test devices are in the GitHub repository but we will give an overview here.
The test suite is supported on various platforms like macOS, Linux, and Windows devices so you can directly install and run the python suite on any of these devices. You can also use the tools on mobile devices but you can’t run the suite directly here,
For single device tests especially, we recommend that you use Virtual Machines (VMs) because of the following advantage:
1 . it’s easy to create and configure new network interfaces
2. you can keep your host machine from third-party software installations that the tests may require
3. it’s easier to capture traffic leaving a VM than to capture traffic leaving a real device
4. you can easily pre-configure snapshots and revert them when necessary
if the testing is serious and to be done in-depth, it’s recommended that you run the test on physical devices as well just to rule out any differences in result between a real device and a virtual one.
Most tests would require root permissions to run so to make things easier, run the whole suite as a root.
Leak Case Study for Test Tools
1 . Case Study 1: Vanilla Leaks
Vanilla leaks are types of VPN leaks that occur when the VPN isn’t protecting the user’s IP address, DNS request, or web traffic under stable working conditions. This type of leak results in privacy and security is severely compromised.
Types of Vanilla Leaks
1 . Public IP Address Leaks
Your IP address becomes visible to websites and apps.
2. DNS Leaks
These are categorized into different classes, each having different severity:
DNS Requests Don’t Go Through The VPN Tunnel but Instead Goes to A Third Party DNS Server
This type of DNS leak is the most severe kind and in this, all DNS requests from a user are visible to hackers and the third party running the DNS server. Since the requests will come from the user’s IP address, it can be directly associated with the individual.
DNS Requests Don’t Go Through The VPN Tunnel but Go to The VPN DNS Server
This is a less severe case of DNS leak compared to the former as the VPN DNS server is trustworthy in most cases. But since the DNS requests are made from outside the VPN tunnel, they are not encrypted and any man in the middle like your ISP would listen in on your DNS requests. Those who are watching will also be able to associate it with the individual through their IP address.
DNS Requests Go Through A VPN Tunnel and Then to A Third Party DNS Server
This is the least severe form of DNS leaks as the DNS requests will be encrypted, preventing any man in the middle from eavesdropping. They won’t be able to directly associate the requests with any individual but in very targeted attacks, an attacker could use some means to get information about the sender.
3. IP Traffic Leaks
In this form of leak, traffic from the device isn’t going through the VPN tunnel. If IP leaks are happening, it also means DNS leaks are happening. If your VPN service experiences such leaks, then it’s the same as not having any VPN connection at all.
Interesting Read : How to Get Unbanned from Omegle and Retrieve Your Account?
Testing for Leaks
1 . Manual Testing
You can easily test for IP leaks and DNS leaks online using the following tools:
The IP leak test doesn’t apply to check for IPv6 leaks so you will have to use the WebRTC leak test for this. The DNS leak test doesn’t verify that the DNS requests went through the VPN tunnel and its more complicated in testing for this.
2. Testing Using LimeVPN Leak Testing Tools
These tools are an extensible suite of open-source python tools that are made for automated and manual testing for VPN leaks. Go here for instructions on download and tools set up. After setting up the tools, be in the tools root directory, and execute the following command:
./run_tests.sh –c
Configs/case_studies/vanilla_leaks.py
The command will run different tests that will check for basic vanilla leaks.
2. Case Study 2: WebRTC Leaks
WebRTC is an open technology that allows web pages to provide P2P communications using a web browser. Many web browsers now support the WebRTC feature.
WebRTC allows web browsers to detect a user’s IP addresses (both public and local) so that P2P can be possible without the need for additional plugins or apps.
WebRTC leaks are just like IP leaks. Web pages can get a user’s IP address from the browser by using JavaScript WebRTC API. Once they have gotten this information, they can send them to their servers and from there identify who is accessing a particular service or page. Using a VPN would protect against IPv4 WebRTC leaks, but they mostly overlook IPv6 address leaks.
How WebRTC Detect IP Addresses
WebRTC utilizes two main techniques to determine location:
STUN/TURN: these servers detect IP addresses by looking out for the public IP that was used to gain access.
Host Candidate Discovery: this is usually the cause of IP leaks and it allows web browsers to have direct access to your system and network interface. This way they can see the IP addresses that are associated with them.
Every active network interface like an active WIFI connection is associated with an IP address so it can function. These IP addresses ordinarily can’t be seen by the web pages but WebRTC reveals them.
Why IPv6 Are Problematic
IPv4 addresses are not enough to go round anymore and so they are shared between people using Network Address Translation. The system allows devices to share common public IPs to assign local IPv4 addresses to their network via the ISP. Since the local IP addresses are used by millions of computers, it’s hard to associate a single address with an individual.
IPv6 addresses are enough to go round and so each individual is assigned one. When using the unique IPv6 address that has been assigned to you, WebRTC can identify you easily even if the address isn’t usually visible to everyone else.
Types of Leaks
There are two types of leaks that a user may suffer from and it depends on the WebRTC related permissions they grant to a webpage. They have different severity and using a good VPN would help protect you in both cases.
1. Persistent Leaks (Requires No Permission)
Web pages don’t have WebRTC permissions by default but it’s still possible for the web pages to gain access to the user’s IP address with WebRTC APIs. When this happens, websites can track a user without the user’s knowledge.
2. Triggered Leaks (Requires Permissions)
Webpages can also ask for permission to be granted so the user can have access to WebRTC related functions. The permissions are for camera and microphone because WebRTC is meant for P2P communications like video calls.
When you grant permissions to any of these features, WebRTC can have access to a wider range of IP addresses on the device, hence a leak. Some browsers don’t ask for permission after the first time as they retain access. It should be noted that different web pages need to be granted permissions individually, so granting access to one doesn’t give access to another.
Even though these leaks are not as severe as the persistent leaks, they are still a mode of leaks. Using a VPN is the only way to hide your IP address even when you have granted permissions to web pages, but if a VPN can’t hide your IP address, you are left with either avoiding the use of WebRTC enabled sites, or accept the risk involved in having your IP address exposed.
Testing for Leaks
These tests for leaks can be performed on any desktop computer and it is recommended that you have an IPv6 address available on the test device as this is the condition in which leaks are common.
1 .Manual Testing
- Disconnect your VPN
- Visit the WebRTC leak test page
- Write down the public IP addresses you see. The addresses would be displayed as “Potential leak”
- Close the test page
- Connect to your VPN
- Reopen the test page
- If the IP addresses are still seen, then you leak.
You must close the test page as instructed to avoid browser caching problems. As many browsers will cache the web page, simply reloading it won’t help with accurate results.
You should note that this test is only valid for persistent leaks.
2. Testing for Leaks with LimeVPN’s Leak Testing Tools
LimeVPN’s leak testing tools are an open-source python program that has been designed to test for leaks both manually and automatically. Get instructions on downloading and setting up the tools here.
After setting up, go to the tools directory and execute the following:
./run_tests.sh -c
Configs/case_studies/webrtc_leaks.py
The command will run tests for persistent leaks without permissions, and also run tests for leaks when permissions are granted.
Case Study 3: Leaks When Switching Networks
Networks are being switched regularly and it happens in different forms:
- Switch from Wi-Fi to mobile data
- Switching between networks
- Switching between Wi-Fi and Ethernet
While these transitions happen swiftly and without any obvious compromise to security in the eyes of the user, the operating systems beg to differ. the changes alter the entire network configuration and can also disrupt VPN application. The alterations can affect the integrity of the protection offered by the VPN leading to leaks.
Leaks from Network Transition
All leaks that come from network transition begin in the same way:
- A network transition occurs
- The integrity of the VPN is shaken
- Leaks occur as the traffic is no longer being routed through the VPN tunnel
Temporary Leaks
Temporary leaks occur during a short time window only, and have the following form generally:
- An issue is detected by the VPN
- The issue is fixed by the VPN
- VPN integrity is restored and the compromise repaired
These types of leaks are not severe and only short-lived, but they are still problematic for the user. They occur regularly and are one of the ways ISPs and other third parties gather information and create a profile on a user.
Persistent Visible Leaks
These types of leaks occur when the user’s VPN detects an ongoing leak but doesn’t fix the issue. It happens sometimes because the application can’t fix the issue or isn’t designed to fix such issues. So what happens is that the VPN would disconnect and leave the user unprotected.
Persistent visible leaks vary in severity and depend on if the user is present when the network transition happens. If the user is present and notice the issue, they can fix it quickly, but if the user is away when it happens, the device will be left unprotected for long. Some VPNs offer a kill switch feature that shuts down internet connection when there are leaks, but the VPN still leaks for a while before the security feature comes into play.
Interesting Read : How to Deactivate or Delete Your Snapchat Account?
Persistent Invisible Leaks
These types of leaks are the most severe as they are undetected by both the VPN and the user. It leads to leaks that lasts for long periods.
Testing for Leaks
In this case study, we would focus on the most severe type of leak; persistent invisible leak. You’ll need a device with two network adapters for this test for both Wi-Fi and Ethernet connections. Ensure that at least one of those adapters is using a DNS server on a local IP.
1 . Manual Testing
The repro steps here are only for macOS as leaks are found to occur mostly on this platform. When testing for other platforms, use LimeVPN Leak Testing Tool.
Checking Prerequisites
- Open the System Preferences and go to Network
- You will see at least two network services on the list bearing green dots next to them. If there are not at least two network services, you won’t be able to run the test.
- The top network is your primary network
- Click on the primary network and then on Advanced
- Look under DNS and ensure that the listed IPs are local. If it isn’t, you won’t be able to run the test. If you are uncertain about your IP being local, use MxToolbox. Enter the IP in the tool and click on Reverse Lookup. If the resultant IP is private and in the form “X.X.X.X”, then it’s local.
Repro Steps
- Disable your primary network by opening “System Preferences” and then go to “Network”. Click on the cog and select “Make Service Inactive”. Then click “Apply”.
- Connect your VPN
- Open the DNS Leak Test page
- Take note of the DNS servers listed
- Enable your primary network
- Refresh the DNS leak page. If DNS is leaking, you’ll see a different list of servers.
2. Testing with LimeVPN’s Leak Testing Tools
LimeVPN’s leak testing tools are an open-source python program that has been designed to test for leaks both manually and automatically. Get instructions on downloading and setting up the tools here.
After setting up, go to the tools directory and execute the following:
./run_tests.sh -c
Configs/case_studies/webrtc_leaks.py
The test will confirm that you have the required prerequisites before running the repro steps to check for DNS leaks.
Case Study 4: Leaks When VPN Server Can’t Be Reached
In some cases, the VPN app can’t connect to the VPN server due to several reasons like:
- Maintenance is ongoing
- Hardware faults
- Blocking
- Bad routes
- Datacenter outages
When any of the above happens, the user will be disconnected from the server. VPN apps would usually take action to restore the connection and prevent any leaks. The time between the reconnection could vary and leaks are unavoidable.
Reachability Leaks
When a VPN server can’t be reached, the apps may behave in the following ways:
- Not notice the issue and leave the user with no internet connection
- Detect the issue and disconnect the user completely from the VPN
- Detect the issue and try to connect the user to a faster server
They all lead to a leak with varying severity.
Testing for Leaks
1 . Manual Testing
It can be difficult to manually test for leaks here as you have no control over the VPN server and can’t simulate a reachability issue easily. Generally, you can simulate a reachability issue by using firewalls on the test device or an upstream device like a router.
Manipulating Firewalls
For different operating systems, we recommend different tools for firewall manipulation:
Windows: Netsh Advfirewall
macOS: Packet Filter
Linux: iptables
Retro Steps
- Connect to a VPN
- Get the VPN’s server address
- Initiate packet capture on every interface using tcpdump for instance
- Create a firewall to block every traffic going to the VPN server IP
- Wait for the block to be noticed by the VPN and stabilize to a new state
- Terminate packet capture
- Disable firewall
- Disconnect from the VPN
- Check traffic capture for packets that left the device unencrypted
2 . Testing Using LimeVPN Leak Testing Tools
LimeVPN leak testing tools are an open-source python program that has been designed to test for leaks both manually and automatically. Get instructions on downloading and setting up the tools here.
After setting up, go to the tools directory and execute the following:
./run_tests.sh -c
Configs/case_studies/webrtc_leaks.py
The tests using the tools will perform the same tests as the manual test above and will create firewalls to block connectivity to the VPN server after determining the server’s IP. It will then monitor leaks for a while and watch how the VPN reacts.
Case Study 5: Leaks When The VPN Process Crashes
Other scenarios have only tested for crashes that occur within the VPN processes and not the UI applications or support binaries that come with the VPNs. It isn’t practical to have a generic test suite that tests crashes of UI applications as the processes are specific to each provider.
Leaks During Crashes
1 . Temporary Leaks
Temporary leaks are those that can occur in a short period and take the following form:
- The VPN will detect a crash
- It would attempt to correct the issue
- The VPN connection would get restored
These leaks are only temporary and less severe as a VPN crash is uncommon and doesn’t expose much of user privacy.
2. Persistent Visible Leaks
These types of leaks happen when the crash is detected but the VPN fails to correct it. It’s sometimes because the VPN app can’t correct the issue, or because the app wasn’t designed to fix the issue. What usually follows is that the VPN application disconnects and the user is left unprotected.
The severity of the leaks varies and depends on if the user is present during the crash. If the user is present, the issue can be fixed promptly, but if not the issue could stay uncorrected for a while.
Some VPN applications have a kill switch and it shuts down internet connection when there are issues that could leave the user exposed. But with such protections, the user’s data still leaks as some time passes before the kill switch becomes active.
3. Persistent Invisible Leaks
In persistent invisible leaks, the VPN doesn’t detect the crash and the user data and connection is left unprotected for a long time. The user would detect this issue as when a VPN crashes the user most times wouldn’t have any internet access.
4. Manual Testing
It’s quite difficult to test for manual crashes. The problem isn’t in simulating the crash of the VPN process, but testing for leaks is tricky and requires packet capture.
Use a tool like tcpdump to capture all outgoing traffic from the network interface. It can simply be done by running the device in a virtual machine and capture all outgoing traffic on the host device.
Simulating A Crash On a Posix Device
Pgrep openvpn | xargs –n1 kill -9
Simulating A Crash On Windows
Taskkill /F /IM openvpn.exe
Repro Steps
- Connect to a VPN
- Initiate packet capture
- Crash the VPN process
- Wait for 2 minutes within which the VPN should have noticed the crash and stabilize
- Stop packet capture
- Disconnect the VPN
- Check the traffic capture for unencrypted requests that left the device
5. Testing for Leak Using LimeVPN Leak Testing Tools
LimeVPN’s leak testing tools are an open-source python program that has been designed to test for leaks both manually and automatically. Get instructions on downloading and setting up the tools here.
After setting up, go to the tools directory and execute the following:
./run_tests.sh -c
Configs/case_studies/webrtc_leaks.py
The command launches the manual tests in an automated way. Packet capture isn’t used to determine if there is a leak but rather they look for DNS leaks and IP leaks. In automated tests, not all packet capture is taken into consideration as the results could lead to false positives.
Conclusion
Data leaks are a very serious issue as they give out your information to anyone who cares to check. It compromises your privacy and makes you susceptible to hackers. Using a VPN can help prevent these leaks but in some cases, VPNs also leak your data. That’s why its important to make use of only the best VPN service provider. LimeVPN is one VPN service provider that works actively to help prevent DNS leaks.
It does this by encrypting your DNS using AES-256 data encryption, and CA 4096 handshake encryption to ensure you are protected. It has a DNS leakage protection feature that works to stop all DNS leaks as it uses its anonymous DNS servers for all DNS requests you make. With good speed and a strict no-logs policy, your privacy is assured.
