I was sitting in the office one day doing my usual checking that everything was okay when I happened to look at the traffic on the firewall and saw the first half of this graph:
Odd outbound firewall traffic. Red is download. Grey is upload.
We do quite a bit of data processing at work and there’s always one big upload or download happening all day long – but the gradual and ever-increasing outbound traffic was just odd. What’s equally as disturbing is that it was worked it’s way up to 45.27Mbps!
Leaping into action (okay, leaping is a bit of a stretch – I was sitting on and took a moment to sip some coffee and wonder what the hell I was looking at, but an opening like that makes for more excitement a-la CSI: SysAdmins.) I checked the pftop output on the firewall and saw that one system on the network was making all of that outbound traffic – specifically the machine of a developer who started with the company recently. Traffic was heading out to an IP address related to a company I recognised and that we could have been sending data to, but this was still really weird.
Deciding to take the easy route first and gather more information, I casually strolled over to her to ask a few simple questions. After all, there’s no reason to panic yet. She was sitting quietly at her desk with earbuds in her ears and quietly coding on her Ubuntu laptop. I walked up and she popped out an earbud – the riveting sounds of talk radio escaping to the ether.
Me: Hey, quick question for you – Are you uploading any really big data files right now?
Her: Hmm? No, I don’t do any of that.
Me: Oh, well I’m seeing a lot of outbound traffic going from your machine to an Akamai address and I just thought I’d check and see if you were doing anything like that.
Her: Do you think I have a virus?
(I love how users immediately jump to the virus conclusion. I said there was no need to panic yet and I meant it – because panic would just make any situation worse.)
Me: I don’t know if I’d say that yet. I just saw this really weird traffic on the firewall coming from your machine and wanted to see if you were doing anything that might explain it. Anything that might be making a bunch of network traffic? Anything that would push out 40 megabits a second?
(I could already by the puzzled look on her face and the contents of her screen that she was doing nothing of the sort. Just Eclipse, FireFox and a filesytem window open. Not even a P2P client in sight.)
Her: Definitely not. I’m just using Eclipse and listening to Bloomberg. Let me close all this stuff.
(Unfortunately, she was too fast and closed all the windows before I could stop her. My hope to prevent panic and not freak her out cost me the he possibility of looking at things I needed to look at on her machine.)
Me: Actually, I was hoping you could keep on doing exactly what you were going so I could take a closer look. I just wanted to make sure I wasn’t looking at any expected traffic.
Her: Oh, okay. So you want me to reboot?
Me: No. Just do everything you would normally be doing – even listening to the radio.
Her: Okay. Let me know what you find.
Me: You’ll be the second to know.
I wandered back to my office already certain that this was going to be an interesting issue. Based on my experience, developers of nefarious software usually want it to go undetected for as long as possible so I’d only expect to see something like this if the ‘virus’ had a bug or was being used to DDoS – and even if it was a DDoS, I’d expect to see a steady stream of data and not an ever-increasing chunk of outbound data.
I watched pftop on the firewall and hoped the strange outbound traffic would show up again and was happy to see it did. It wasn’t at the crazy-high levels I was seeing before she killed everything, but it was there and it was actively increasing. I started a packet capture on the firewall to do a quick grab of all traffic going between her machine and the destination server.
Opening the file of captured packets in Wireshark, I saw a large number of RTMP packets going back to the server. Right-clicked on one, selected Follow TCP Stream to reconstruct what was going on and was presented with a very interesting and large window of data that read:
POST /idle/2iMmdz02wSLmyUHI/8097 HTTP/1.1 Host: 96.6.41.39:1935 Accept: */* User-Agent: Shockwave Flash Connection: Keep-Alive Cache-Control: no-cache Content-Type: application/x-fcs User-Agent: Shockwave Flash Connection: Keep-Alive Cache-Control: no-cache Content-Type: application/x-fcs User-Agent: Shockwave Flash Connection: Keep-Alive Cache-Control: no-cache Content-Type: application/x-fcs User-Agent: Shockwave Flash Connection: Keep-Alive Cache-Control: no-cache Content-Type: application/x-fcs <... repeat the above 4 lines *many* times over ...> User-Agent: Shockwave Flash Connection: Keep-Alive Cache-Control: no-cache Content-Type: application/x-fcs Content-Length: 1
What it looked like was happening was the Adobe Shockwave Flash component that was streaming her Bloomberg Radio audio was sending back keep-alive packets to tell the server to keep sending the audio stream. This is actually normal behaviour, except the rate at which it was doing this was increasing as time passed. I assume it would continue this unfortunate pattern of screaming back to the server until it was either interrupted or it completely saturated all upstream bandwidth. Only one way to find out.
I set up a test environment in my lab to isolate the systems and replicate the situation. Same firewall. Simplified network of a single client machine. Basic installation of Ubuntu Linux 11.04. A visit to http://bloomberg.com/radio under Firefox and pressed play on the audio player. The results were pleasantly identical.
Listening to Bloomberg Radio in the lab uploaded over a DVD's worth of data in less that two hours.
Although inbound traffic never exceeded 226kbps and total content downloaded for the test period was 124.2MB, the outbound traffic gradually built up to 9.67Mbps with a total upload of 4.69GB in about 1h49m. The stream self-terminated and ended the test prematurely, but enough data had already been collected to confirm a theory: Bloomberg Radio was killing my upstream bandwidth and I was very glad I wasn’t paying for upstream bandwidth by volume.
Continuing to test on several different operating systems and browser combinations I found that Windows and Mac were completely unaffected by this issue and the problem itself seems limited to the Bloomberg Media Player version 1.0428 running with Adobe Shockwave Flash Player version 10,3,183,5 under Linux. Browser choice did not seem to matter as they all utilise the same libflashplayer.so plugin file from Adobe to execute the audio stream flash object.
Since discovering this odd bug, the problem has continued to show up day after day. I explained to the user what I found and she’s made a conscious effort to pause the audio stream when she gets up from her desk and in return I have been nice and didn’t set any rules to block her audio stream. Know someone at Bloomberg that can get this big fixed? I’d appreciate you forwarding this over to them.
