Yesterday around noon, the internet at my company started acting up. No matter, slowdowns happen and there’s roadwork going on outside: maybe they hit the fiber or something. So we waited.
Then our Samba servers started getting flaky. And the database too. Uh oh… That’s different.
We started investigating. Some machines were dropping ICMP packets like crazy, then recovered, then other machines started to become unpingable too. I fired up Wireshark and discovered an absolute flood of IGMP packets on all the trunks, mostly broadcast from Windows machine. It was so bad two Linux machines on the same switch couldn’t ping each other reliably if the switch was connected to the intranet.
So we suspected a DDOS attack initiated from within the intranet by an outside attacker. We cut off the internet, but the storm of packets kept on coming. Physically disconnecting machines from the intranet one by one didn’t do a thing either.
Eventually, we started disconnecting each trunk one by one from the main router until we disconnected one and all the activity lights immediately stopped on all the ports. We reconnected it and the crazy traffic resumed.
So we went to that trunk’s subrouter and did the same thing. When we found the cable that stopped all the traffic, we followed it and finally found one lonely $10 ethernet switch with… a cable with both ends plugged into the switch. We disconnected the cable and everything instantly returned to normal.
One measly cable brought the entire company to a standstill for hours! Because half of the software we have to use are cloud crap or need to call their particular motherships to activate their licenses, many people couldn’t work anymore for no good technical reason at all while we investigated the networking issue.
Anyway, I thought switches had protections against that sort of loopback connection, and routers prevented circular routes. But there’s theory and there’s reality. Crazy!
Does that kind of loop really mess with things? ELI5 please!
Also, what do you mean a lonely switch? Does it have that loop and a port connected to another switch in the network?
IT tech here, yes, yes it can.
Network infrastructure is both increadibly smart while also being dumb in other ways.
To do an ELI5 answer:
Imagine you have a container of pearls that you need to sort, red, green and blue pearls all need to be dropped into a red, green or blue hole.
The container is being refilled, but slow enough that it only gets a new pearl once you have sorted the previous.
The holes are connected to pipes going to separate buckets.
Everything is fine, but then some adds a new hole that is muticolored and tells you that all pearls should go there.
You tell your friends that you have a faster way to deal with the perls and to send you their pearls.
The new hole also has a pipe, but that is connected to the container that recieves pearls, so every time you drop a pearl into the new hole, it appears in the container again.
So now you have a situation where you not only get your normal ammount of pearls, but everyone else’s pearls and you also get every pearl you send back again.
You are smart and quickly realize that something is wrong and call for your teacher for help, networking gear don’t have that capabillity to understand that it is wrong, it just looks at each pearl and not the big picture.
If we go back to the real world, we have developed tools to deal with this situation, we have protocols line spanning tree which can have switches speak with eachother and figure out if there is a physical loop before sending traffic through it.
There are other tools as well, but they all need to be configured and to be honest, it is easily forgotten or made a low priority since it happens rarely.
It is something that is often implemented after a big outage.
Certain types of broadcast traffic always get re-broadcast from of every port on a switch. So if you directly connect two ports, and you get some broadcast coming into the switch, that broadcast will loop forever across that loopback, and then get propagated repeatedly until it hits a broadcast boundary. It’s surprisingly difficult to prevent even with managed switches unless you are willing to hand manage every port and significantly restrict the kind of network services which can flow through it.
Some devices can detect these loops and break them, but that can have other unintended impacts if your network is designed (some would argue poorly) around using dumb switches to multiply limited Ethernet drops at the edge.