Saturday, December 20, 2014

Fixing The Internet, Part 1 – Spiky Noise

The last three months we’ve been focusing 100% on developing our product, building our own firmware for Jensens Air:Link 7000ac that enable logging of key data to our cloud. We were extremely curious as to what we would find, and we have not been disappointed. In fact, we found so much wickedness that we changed our mission to nothing less than fixing the internet...

I’ll share some examples in a series og blog posts, first up one on radio noise.

This happened one early Monday morning late November. Our cloud diagnostics algorithm suddenly started screaming. Our Oulu office router was logging noise data that almost went through the roof. What the heck was going on?

I started diving into the data and found radio noise levels on the 5GHz band up towards -50 db. This is 40 db over the noise floor, or approx 10,000 times the normal noise level. Noise at these levels basically kill the network.
Interesting radio noise
What was really interesting was the pattern. Noise normally establish a new noise floor, but now we saw an unusual spike characteristic. What could be the source of the pollution?

 We jumped on call with Oulu and learned that Jukka, our new Linux wizard, had received his new Lenovo laptop that morning. And the router was placed on top of the laptop. So fair assumption was that the laptop was the culprit, but why the spikes?

 Some more testing and many creative theories later we nailed it. It was the power supply. But the catch was, it was only polluting when the laptop battery was charging! Good luck figuring that out without reliable diagnostics.
Router on laptop with power supply nearby
Good news is that this type of noise characteristics can be classified and identified using Machine Learning techniques. And typically the noise only cover a narrow frequency band, and pollute only over short distance. So it is easy to fix, that is if you are able to identify the problem.


 In the next post we’ll share some super interesting learnings on latency and buffer-bloat. Stay tuned!

Posted by at

2 comments:

  1. Dave TahtDecember 21, 2014 at 9:38 PM

    If you would like someone to review a preprint of your upcoming bufferbloat piece feel free to contact me or those on the bloat mailing list.

    A good tool to look at bufferbloat related problems is the netperf-wrapper rrul test suite.

    If your device is linux derived, the sqm-scripts (with fq_codel) are capable of tuning up your isp up and downlink to have maximum throughput with minimum latency. The results of a typical tuning session for a cable modem:

    http://burntchrome.blogspot.com/2014_05_01_archive.html

    If you want to see the horribleness that most 802.11ac drivers have in them, try the netperf-wrapper rtt_fair4be test to 2 or more stations at the same time.

    ReplyDelete
    Replies
    1. UnknownDecember 22, 2014 at 4:20 AM

      Thanks Dave, appreciate that. Will IM you. We have applied traffic shaping based partly on inspiration from you guys at Bufferbloat project, and in many cases see the typical result. We do however see significant variations that we guess stem from upstream network topology, so we try to learn and dynamically optimize for each individual router.

      Delete

Subscribe to: Post Comments (Atom)