【NetEyez Use Case #1】Delay in Internet Communication (Part 1)
〈Incident Description〉
Since the beginning of the year, Company A has experienced delays in internet access.
Users encounter longer loading times when accessing websites on the internet, and disruptions in audio or distorted images frequently occur during web conferences. The incident has been persistently happening during business hours.
〈Company A’s Network Environment〉
In Company A’s network, internet communication traffic flows through a proxy server installed within the headquarters, reaching the internet via the internet gateway.
Remote users outside the office are accommodated in the company’s network through a VPN gateway (see Figure 1).
Company A has deployed four proxy servers, and traffic is distributed based on communication content. General web access traffic is processed through Proxy1 and Proxy2, while Microsoft 365 traffic, including email and web conferencing, bypasses Proxy3 and Proxy4 for load balancing. The internet gateway has a line speed of 100Mbps, accommodating internet communication for 600 users.
Coinciding with the government’s declaration of a state of emergency due to the widespread of the novel coronavirus, Company A has been implementing restrictions on in-office work since the beginning of the year and actively promoting telecommuting for its employees.
〈Cause Investigation〉
Upon checking the traffic statistics of the internet gateway using the MRTG tool, it was observed that the downstream traffic (from the internet to Company A) reaches the bandwidth limit (100Mbps) during certain periods but does not consistently remain congested (see Figure 2).
In the Proxy1 and Proxy2 systems, upon checking the thread count information, it was identified that during the non-lunch hours of business operation, the thread count has reached the maximum limit of 1,500 threads (see Figure 3).
Connecting the network monitoring tool NetEyez to the server switch (see Figure 1) and monitoring web traffic revealed a significant bias in traffic towards Proxy1 and Proxy2 (see Figure 4).
Furthermore, examining the average response times of communications through these proxy servers, it was observed that the response times of Proxy1 and Proxy2 significantly exceeded those of Proxy3 and Proxy4 (see Figure 5).
Checking the IP addresses of users accessing Proxy1 and Proxy2, it was identified that a significant number of users were accessing via VPN (see Figure 6).
In contrast, access to Proxy3 and Proxy4 did not include remote users, and access was limited to users within the office.
〈Solution〉
The Cause was the Sudden Increase in Remote Users
The root cause of the issue was the sudden increase in remote users.
All traffic from remote users was being directed to Proxy1 and Proxy2, regardless of communication content. This resulted in an imbalance in the load on these two proxy servers, causing delays in internet access communication.
↓
Modifications of VPN Gateway Settings
To address this, modifications were made to the VPN gateway settings to distribute the traffic of remote users across all four proxy servers. This alleviated the overload condition on Proxy1 and Proxy2, leading to an improvement in internet communication delays.
NetEyez Use Case List
Click here for details about NeEyez.