Fixing The Internet, Part 2 - Bufferbloat

If you ask the average user, quality of internet connections are measured in bandwidth. Period. Not surprising really, as this is what has been communicated by the industry. If your internet is slow, you just need to upgrade your bandwidth – at a premium of course.

But there is another metric that is just as important to the quality of the connections, and that is latency. Latency (or lag, ping time) is the time it takes for a tiny packet of data to be sent through the network. This is critical in real time response use cases as telephone or video conferencing, online gaming and normal browsing. In theory, the latency should just be a function of how far away the content is, i.e. how many nodes it needs to travel through. In reality, it is a function how buffers have been configured and managed in the network.

Buffers are queues that have been configured to make the connections more robust. Sometimes a node gets busy. If there were no buffers, packets would be lost. So to maximize bandwidth, the ISP might increase buffer size to make sure that all packets will be delivered.

For video streaming, this is great. It doesn't matter if the stream is delayed a few seconds, so better buffer up and make sure everything comes through. The problem is that the buffers are mostly managed as a simple FIFO queue. Nothing is prioritized, and everything gets stuck in the same queue.

Our tests show that ISP's and networks have very different buffer policies and management algorithms. Some are managing OK, some get into trouble when uploading but most run into problems on downloads – simply because the ISP has capped the download capacity on the connection.

\What happens is that when you request content from some internet server, the internet server will try to send it to you as quickly as it can. It doesn't know about the cap. So when the ISP cap the connection, some of the content will need to wait in a buffer. And everything else will need to wait as well.

Latency while streaming YouTube

The diagram above is a typical result. When traffic is unmanaged, latency is a function of download rate. Latency at 300ms or more will have significant impact on the quality of telephony, gaming and browsing.

In the later part of this experiment, we applied a traffic shaping algorithm on the router that put a constraint on the bandwidth for the device to 20Mbps (or 80% of the capacity of the 25Mbps fiber connection in this example). As the result shows, latency is almost eliminated.

What is really interesting here is that the YouTube video only needed 5Mbps, or 20% of the download capacity. So why did we still get the latency issues? YouTube and others are using a dynamic streaming algorithm called MPEG-DASH. It will only keep a limited buffer in the device, maxing out on the download for a few seconds and then wait until the client is catching up. So even if the video only needs a fraction of the available bandwidth, it makes the internet stutter for all other users of the connection!

We automatically test bandwidth and buffering on a daily basis on our routers. Through this we can learn what is the optimal traffic shaping policies, and dynamically configure the router accordingly. Through this we have been able to remove 70-80% of the latency issues we have observed so far. Will share more on this in a follow up post.

Thanks to Dave Taht for adding insights to this post.

Fixing The Internet, Part 1 – Spiky Noise

The last three months we’ve been focusing 100% on developing our product, building our own firmware for Jensens Air:Link 7000ac that enable logging of key data to our cloud. We were extremely curious as to what we would find, and we have not been disappointed. In fact, we found so much wickedness that we changed our mission to nothing less than fixing the internet...

I’ll share some examples in a series og blog posts, first up one on radio noise.

This happened one early Monday morning late November. Our cloud diagnostics algorithm suddenly started screaming. Our Oulu office router was logging noise data that almost went through the roof. What the heck was going on?

I started diving into the data and found radio noise levels on the 5GHz band up towards -50 db. This is 40 db over the noise floor, or approx 10,000 times the normal noise level. Noise at these levels basically kill the network.

Interesting radio noise

What was really interesting was the pattern. Noise normally establish a new noise floor, but now we saw an unusual spike characteristic. What could be the source of the pollution?

We jumped on call with Oulu and learned that Jukka, our new Linux wizard, had received his new Lenovo laptop that morning. And the router was placed on top of the laptop. So fair assumption was that the laptop was the culprit, but why the spikes?

Some more testing and many creative theories later we nailed it. It was the power supply. But the catch was, it was only polluting when the laptop battery was charging! Good luck figuring that out without reliable diagnostics.

Router on laptop with power supply nearby

Good news is that this type of noise characteristics can be classified and identified using Machine Learning techniques. And typically the noise only cover a narrow frequency band, and pollute only over short distance. So it is easy to fix, that is if you are able to identify the problem.


In the next post we’ll share some super interesting learnings on latency and buffer-bloat. Stay tuned!

New investor - Norwegian Founders Fund

We are delighted to announce our new investor, the Norwegian Founders Fund. It is an early stage tech fund set up by a group of Norwegian tech entrepreneurs, headed by Opera Software’s Jon Tetzchner.  So once again, we are fortunate to get financial backing from a group of individuals that are also fantastic advisers.

We are now heads down in product development. Things are going very well, and the ambition of launching our pilot by end of year seems within reach. The first version will focus on “intelligent wifi” features, essentially a smart wifi router that can learn and adapt on its own. We will be able to offer a radically simplified user experience that provide deep insight and control of the home wifi network.

With Founders Fund backing, we will have a product ready in just a few months. Stay tuned!

The End of Smart Devices

We are quickly moving towards the point where connectivity is not defining a new category of devices. It will just be a value-adding feature to the existing.

Tony Fadell made this point implicitly a few weeks ago, when rejecting the notion of Nest as an Internet-of-Things company. In his mind, they just leverage technology and design to make better thermostats and smoke sensors.

While the number of crowdfunded “smart devices” is growing by the day, they have yet to become relevant to the mass market. Yes, Nest have sold more than a million thermostats, but how many thermostats are there in US? Fact remains that the single breakout success of “smart devices” is still in the low single-digits in terms of market share.

The cost of enabling an appliance with wi-fi and/or Bluetooth Smart is now down to a few dollars. Hardware cost will soon be almost insignificant. Question is, what primary use case will it solve?

We believe the simple answer is service and support. If your fridge connect to the internet, the manufacturer or retailer can diagnose and fix issues from remote. Manufacturers think returns may decrease by 50%. That is huge. And this deep serviceability will drive mass market adoption.

When connectivity is a feature and not a product, the connected home will grow out of its current hype cycle and mature into holistic service-and-support focused platforms. We will no longer have a category called “smart devices”. We will just have devices that are smart. Consumers will engage with device directly over mobile apps, but managed through a “home control panel” that sits on the wi-fi router or on the cloud.

The winning formula will be to integrate existing serviceability channels and value chains into a frictionless and intuitive experience for the end consumer.

Maker Faire Silicon Valley

Spent the afternoon yesterday at Maker Faire Silicon Valley in San Mateo. Was just blown away of all the awesomeness. Some pictures:

Beer Robot - the perfect bro gift

... alternatively, the mind controlled beer machine? Didn't get to test it unfortunately

Classic

Sony MESH project - small cubes with BLE sensors for the home. Very interesting...

Sphero - robot balls starting at $99 per set

Fog Computing by Cisco

Launching a new buzzword is often a cheap attention-grabber in the valley, but Cisco's recent announcement of their "Fog Computing" fabric model for Internet of Things actually makes great sense and is a real innovation. As it happens, it is also a precise description of Domos Labs philosophy and architecture.

"Fog Computing" combines "cloud" and "edge" computing - i.e. leveraging the power of cloud through local service nodes, on the "edge". In practice it means cloud-defined, intelligent networking devices that provide smart interfacing and monitoring of devices connected to the edge. Sounds familiar?



Cisco IOx / Fog Computing architecture (linked to Source)

To facilitate their "Fog" model, Cisco will extend their established network device software stack (IOS) with a parallel Linux stack. In this they will allow partners and device makers to build in their own device interfaces and even host their open applications, on the network device. Again this is exactly parallel to our thinking, allowing vertical apps to host processes in our Linux-driven device middleware.

Cisco only target industrial use cases for IoT. But given the shared vision for this type of computing fabric there could be synergies in collaboration - building joint API/sandbox model for developers and device manufacturers for instance. Many devices and use cases will be similar. Initial dialogue is positive so looking forward to explore this further with the team.

Anyway, kudos to Cisco and their IoT team for coining and nailing this!

Domos Labs website (re)launched

Our website got a well overdue update today, finally communicating the concept and value offering we have iterated to over the last few months - check it out here. Funny how changes happen so gradually that you don't really notice unil taking a step back and seeing how much has changed over the last six months.

The website now communicate the new cloud+device software offering, focusing on our target customers (service providers and OEMs). Also change of logo and name, we will stick with Domos Labs to avoid name confusion and shark fights.

So our new key visual is the Cloud+Device infrastructure below. The added focus on cloud capabilites will enable some super exciting use cases involving predictive analytics and machine learning. Still getting our heads around what could be possible.

We have had great response on this when engaging with prospect customers. Easy to get meetings, and people seem genuinly intrigued by our vision and concept, so we should be able to sign up a few pilot partners before our Demo Day May 1st.