Fixing The Internet, Part 2 - Bufferbloat

If you ask the average user, quality of internet connections are measured in bandwidth. Period. Not surprising really, as this is what has been communicated by the industry. If your internet is slow, you just need to upgrade your bandwidth – at a premium of course.

But there is another metric that is just as important to the quality of the connections, and that is latency. Latency (or lag, ping time) is the time it takes for a tiny packet of data to be sent through the network. This is critical in real time response use cases as telephone or video conferencing, online gaming and normal browsing. In theory, the latency should just be a function of how far away the content is, i.e. how many nodes it needs to travel through. In reality, it is a function how buffers have been configured and managed in the network.

Buffers are queues that have been configured to make the connections more robust. Sometimes a node gets busy. If there were no buffers, packets would be lost. So to maximize bandwidth, the ISP might increase buffer size to make sure that all packets will be delivered.

For video streaming, this is great. It doesn't matter if the stream is delayed a few seconds, so better buffer up and make sure everything comes through. The problem is that the buffers are mostly managed as a simple FIFO queue. Nothing is prioritized, and everything gets stuck in the same queue.

Our tests show that ISP's and networks have very different buffer policies and management algorithms. Some are managing OK, some get into trouble when uploading but most run into problems on downloads – simply because the ISP has capped the download capacity on the connection.

\What happens is that when you request content from some internet server, the internet server will try to send it to you as quickly as it can. It doesn't know about the cap. So when the ISP cap the connection, some of the content will need to wait in a buffer. And everything else will need to wait as well.

Latency while streaming YouTube

The diagram above is a typical result. When traffic is unmanaged, latency is a function of download rate. Latency at 300ms or more will have significant impact on the quality of telephony, gaming and browsing.

In the later part of this experiment, we applied a traffic shaping algorithm on the router that put a constraint on the bandwidth for the device to 20Mbps (or 80% of the capacity of the 25Mbps fiber connection in this example). As the result shows, latency is almost eliminated.

What is really interesting here is that the YouTube video only needed 5Mbps, or 20% of the download capacity. So why did we still get the latency issues? YouTube and others are using a dynamic streaming algorithm called MPEG-DASH. It will only keep a limited buffer in the device, maxing out on the download for a few seconds and then wait until the client is catching up. So even if the video only needs a fraction of the available bandwidth, it makes the internet stutter for all other users of the connection!

We automatically test bandwidth and buffering on a daily basis on our routers. Through this we can learn what is the optimal traffic shaping policies, and dynamically configure the router accordingly. Through this we have been able to remove 70-80% of the latency issues we have observed so far. Will share more on this in a follow up post.

Thanks to Dave Taht for adding insights to this post.

Fixing The Internet, Part 1 – Spiky Noise

The last three months we’ve been focusing 100% on developing our product, building our own firmware for Jensens Air:Link 7000ac that enable logging of key data to our cloud. We were extremely curious as to what we would find, and we have not been disappointed. In fact, we found so much wickedness that we changed our mission to nothing less than fixing the internet...

I’ll share some examples in a series og blog posts, first up one on radio noise.

This happened one early Monday morning late November. Our cloud diagnostics algorithm suddenly started screaming. Our Oulu office router was logging noise data that almost went through the roof. What the heck was going on?

I started diving into the data and found radio noise levels on the 5GHz band up towards -50 db. This is 40 db over the noise floor, or approx 10,000 times the normal noise level. Noise at these levels basically kill the network.

Interesting radio noise

What was really interesting was the pattern. Noise normally establish a new noise floor, but now we saw an unusual spike characteristic. What could be the source of the pollution?

We jumped on call with Oulu and learned that Jukka, our new Linux wizard, had received his new Lenovo laptop that morning. And the router was placed on top of the laptop. So fair assumption was that the laptop was the culprit, but why the spikes?

Some more testing and many creative theories later we nailed it. It was the power supply. But the catch was, it was only polluting when the laptop battery was charging! Good luck figuring that out without reliable diagnostics.

Router on laptop with power supply nearby

Good news is that this type of noise characteristics can be classified and identified using Machine Learning techniques. And typically the noise only cover a narrow frequency band, and pollute only over short distance. So it is easy to fix, that is if you are able to identify the problem.


In the next post we’ll share some super interesting learnings on latency and buffer-bloat. Stay tuned!

New investor - Norwegian Founders Fund

We are delighted to announce our new investor, the Norwegian Founders Fund. It is an early stage tech fund set up by a group of Norwegian tech entrepreneurs, headed by Opera Software’s Jon Tetzchner.  So once again, we are fortunate to get financial backing from a group of individuals that are also fantastic advisers.

We are now heads down in product development. Things are going very well, and the ambition of launching our pilot by end of year seems within reach. The first version will focus on “intelligent wifi” features, essentially a smart wifi router that can learn and adapt on its own. We will be able to offer a radically simplified user experience that provide deep insight and control of the home wifi network.

With Founders Fund backing, we will have a product ready in just a few months. Stay tuned!

The End of Smart Devices

We are quickly moving towards the point where connectivity is not defining a new category of devices. It will just be a value-adding feature to the existing.

Tony Fadell made this point implicitly a few weeks ago, when rejecting the notion of Nest as an Internet-of-Things company. In his mind, they just leverage technology and design to make better thermostats and smoke sensors.

While the number of crowdfunded “smart devices” is growing by the day, they have yet to become relevant to the mass market. Yes, Nest have sold more than a million thermostats, but how many thermostats are there in US? Fact remains that the single breakout success of “smart devices” is still in the low single-digits in terms of market share.

The cost of enabling an appliance with wi-fi and/or Bluetooth Smart is now down to a few dollars. Hardware cost will soon be almost insignificant. Question is, what primary use case will it solve?

We believe the simple answer is service and support. If your fridge connect to the internet, the manufacturer or retailer can diagnose and fix issues from remote. Manufacturers think returns may decrease by 50%. That is huge. And this deep serviceability will drive mass market adoption.

When connectivity is a feature and not a product, the connected home will grow out of its current hype cycle and mature into holistic service-and-support focused platforms. We will no longer have a category called “smart devices”. We will just have devices that are smart. Consumers will engage with device directly over mobile apps, but managed through a “home control panel” that sits on the wi-fi router or on the cloud.

The winning formula will be to integrate existing serviceability channels and value chains into a frictionless and intuitive experience for the end consumer.

Maker Faire Silicon Valley

Spent the afternoon yesterday at Maker Faire Silicon Valley in San Mateo. Was just blown away of all the awesomeness. Some pictures:

Beer Robot - the perfect bro gift

... alternatively, the mind controlled beer machine? Didn't get to test it unfortunately

Classic

Sony MESH project - small cubes with BLE sensors for the home. Very interesting...

Sphero - robot balls starting at $99 per set

Fog Computing by Cisco

Launching a new buzzword is often a cheap attention-grabber in the valley, but Cisco's recent announcement of their "Fog Computing" fabric model for Internet of Things actually makes great sense and is a real innovation. As it happens, it is also a precise description of Domos Labs philosophy and architecture.

"Fog Computing" combines "cloud" and "edge" computing - i.e. leveraging the power of cloud through local service nodes, on the "edge". In practice it means cloud-defined, intelligent networking devices that provide smart interfacing and monitoring of devices connected to the edge. Sounds familiar?



Cisco IOx / Fog Computing architecture (linked to Source)

To facilitate their "Fog" model, Cisco will extend their established network device software stack (IOS) with a parallel Linux stack. In this they will allow partners and device makers to build in their own device interfaces and even host their open applications, on the network device. Again this is exactly parallel to our thinking, allowing vertical apps to host processes in our Linux-driven device middleware.

Cisco only target industrial use cases for IoT. But given the shared vision for this type of computing fabric there could be synergies in collaboration - building joint API/sandbox model for developers and device manufacturers for instance. Many devices and use cases will be similar. Initial dialogue is positive so looking forward to explore this further with the team.

Anyway, kudos to Cisco and their IoT team for coining and nailing this!

Domos Labs website (re)launched

Our website got a well overdue update today, finally communicating the concept and value offering we have iterated to over the last few months - check it out here. Funny how changes happen so gradually that you don't really notice unil taking a step back and seeing how much has changed over the last six months.

The website now communicate the new cloud+device software offering, focusing on our target customers (service providers and OEMs). Also change of logo and name, we will stick with Domos Labs to avoid name confusion and shark fights.

So our new key visual is the Cloud+Device infrastructure below. The added focus on cloud capabilites will enable some super exciting use cases involving predictive analytics and machine learning. Still getting our heads around what could be possible.

We have had great response on this when engaging with prospect customers. Easy to get meetings, and people seem genuinly intrigued by our vision and concept, so we should be able to sign up a few pilot partners before our Demo Day May 1st.

Why Americans don't trust their ISP

Kansas city is one of three areas that are served by Google Fiber, one of very few Fiber-To-The-Home (FTTH) initiatives in US. Subscribers on average get double bandwidth for the same price. One would think this would be appreciated as a good thing, but the state legislators seems to see things differently. Or rather, their puppeteers do.

I recently spent 7 years in China, and thought I had experienced my share of corrupt government. But the Chinese at least tried to maintain a façade of who were actually calling the shots.

Why does this matter? US is already far behind Europe and Asia in residential broadband, and the gap is widening. Anecdotally, I get better broadband at lower cost in my remote French Alpe village than I do in the heart of Silicon Valley.


Netflix Country Average Bandwidth

In Europe, EU is putting FTTH front and center of their Europe 2020 strategy. In US, residential fiber connections seems not to be on the agenda at all. The industry experts I meet have all but given up hope, saying this is business as usual. The Cable Lobby will routinely stop initiatives through local legislation when necessary.

It is really quite sad to see them wasting their resources on buying politicians instead of investing in better services. Makes me question if the ISP channel strategy we are establishing in Europe really can be replicated in US, or if we need to change approach...

Google Getting in the Game

Nest valuation at more than $3B is huge in itself, but Google cashing it out makes it a landmark. This was the day the big guys joined the game. Really looking forward to what will come out of this, especially keen to see Google Now applied to appliances and sensors.

We're prepping for an outreach to service providers, will be super interesting to learn what effect this acquisition will have on their priorities. We have already been using the potential threat of Google entering this market as a strategic call to action. Now the threat is real.

Google is almost religiously a cloud company. They will do hardware and devices as a source of user interaction and data for the cloud. The Google antidote is cable companies and their industry alliance that build "smart" into the set top box.

Our focus on re-inventing Wi-Fi router puts us somewhere in the middle, and also offer the ISPs a competitive and defendable position versus cloud competitors. Sure feels like we're at the right place at the right time...

Bitcoinia

At first I thought Bitcoin was silly - geek libertarian naïveté of using technology to escape the forces of evil called “financial system” and “government control”. And don’t get me wrong - a platform sporting killer apps like “Drug Dealing for Dummies”, “MoneyLaundry 2.0” and the unexpected Chinese runaway success of “From China with Cash” (which has gone largely unnoticed in the US), expecting governments to keep their hands off? Good luck with that.

I was surprised seeing an authority like Chris Dixon leading $25M investment in Coinbase. Earlier investments in the space had been somewhat on the other end of the gravitas scale. Was I missing something?

 I took the time to dig a bit deeper into the technology and protocol, and actually agree that this is significant and disruptive innovation. It has the potential to make money transfers and micropayments way more secure and efficient than what is available in US today. But then, US is also hopelessly lagging the rest of the world in this, as in other infrastructure investments.

Both Europe and China already have national payment gateways, through collaboration between financial institutions and the governments. They are non-profit. Government provide regulatory, banks provide oversight. The result is super efficient debit plastic, with fixed transaction fee of a few cents. It also allows such “radical” ideas as direct wire transfer between accounts in different banks, without having to write a check. To European and Asian readers: yes, you read that right.

While visiting the Nordics I have not needed cash for years. Plastic is accepted everywhere, at no significant cost for anyone. In the occasion when you need to pay a friend or split a bill, a mobile banking transfer is now as easy as sending a text. China is quickly moving in the same direction through UnionPay, which might transform the global credit card business.

But Bitcoin could surely solve real use cases, also outside US. Social, digital micro-incentives in advertising and gaming is a good example. I am sure there are many more.

Still, Bitcoin in its current form as a currency is doomed. Why? Because a currency needs stability, and stability is provided by a central bank.

To enthusiasts, the idea of having a currency with no central bank seems to be a good thing. The argument is along the lines that it will become more stable as central banks cannot inflate the value by printing more money to fund government deficit.

So here’s my prediction of how Bitcoin will play out in the current form. I believe the currency will survive this first hype curve, supported by enthusiasts in tech community. What will ultimately kill it as a currency is when increased transaction volume makes it a meaningful asset class for hedge funds. With no backing or oversight, it will be paradise for market making traders. They make money whenever there is volatility and sufficient volume of amateur traders to crush. The resulting permanent volatility will effectively make Bitcoin unusable as a currency.

Bitcoin could work great though if it were just a digital payment protocol. If it was pegged to the dollar, or rather a basket of dollar, euro, yuan and yen, it could be a serious disruption and help make financial transactions frictionless. Digital exchanges like Coinbase could become the new de-facto digital payment gateway. Could this be the scenario the smart money is investing in?

Suits vs Surfies

Interesting to observe the micro-cultures of the valley, almost as diverse as the microclimates. I have spent last couple of weeks getting involved with the Telecom community, centered down south in San Jose. They talk about semiconductors and Moores law - actual substance for the real world. But slow-moving.

The urban hipster haven of SoMa feels worlds apart from San Jose.  SoMa talk is Insta and Snap. Lightweight, viral entertainment. And counter-intuitive to the microclimate – hot and sunny San Jose dress code is suit, while cold and foggy city wear is surfer shorts.  

And then there is us in the middle. We are somewhere on the Peninsula – San Mateo, Palo Alto, Mountain View.  We do software and internet and cloud stuff. We dismiss city as fluff and shrug at San Jose as boring hardware.

City is not all fluff of course, and hardware is far from dead or boring. So many new exciting device+service products launched this year, and innovations of Arduino and RasPi making creative process so much more dynamic and accessible. But Telco world keep pitching dead-on-arrival technologies like NFC for mobile payments. As our Lean Startup mentors teach, they need to learn how to fail faster...

Angels for Christmas

Just a quick update that we recently closed an angel investment round, raising more than $100k. In addition we won a $50k grant from Innovation Norway.

.

Our angel investors are a great bunch of experienced industry experts with heaps of clout and acumen. Charles Huang and Kai Huang bootstrapped Guitar Hero. Anders Brandt built a Norwegian internet service provider. WilliamKlippgen co-founded Kelkoo. Bjørn Haugland built ConfirmIt to a global software company. Chris Evdemon is insider of Silicon Valley angel and VC community.

This is our Dream Team. And as with any Christmas present, it is really the thought that counts. We are very proud to have earned your confidence and trust!

In our own backyard

Sometimes you go casting a wide net only to find what you look for in your own backyard.

We where in the need of a good developer that had experience in OpenWRT, a Linux-based software that can be used on a range of routers. Starting with Google, this lead us to a Master Thesis written about this in the southern part of Norway some months ago. Then talking to the one writing this paper then lead us to yet another person that he had used as an advisor.

We then engaged in conversations with this person, and understood that this might be the right guy for us. And where was he based? In the room next to us in StartupLab here in Oslo.

I could have literary jumped on top of my desk, shouted ”Any OpenWRT-knowledge here?” and he would have been able to answer.

Employee No. 1 is about to get hired!

Becoming Alchemists

Proud to announce that we’ve been selected for next batch of Alchemist Accelerator in Menlo Park. It is a six month program backed by leading VCs and is only for B2B startups. Heard lots of great stuff on how Ravi Belani and his team run the program, so really looking forward to it.

Some other big news coming up as well, more updates next week 

Urban Mill

Urban Mill / Startup Sauna outside Helsinki is definition of new cool. Laidback, anarchistic, hipster incubator lab next to Aalto, the major tech university. Anyone can get in and hang around, desks are available on a “first come first served” basis. Old factory buildings, high roof and no walls. Open space – in every possible meaning.

Active Life Village in Urban Mill, Helsinki

We knew immediately that this was a perfect location for one of our smart home labs. Urban Mill is defined as a “co-working and co-creation platform prototype for urban innovation”, sporting several initiatives that fit perfectly with ours. ActiveLife Village is all about wellbeing for senior citizens, lots of wireless home solutions and technology. Asema is doing great stuff in smart energy management. We can create a really cool cluster of companies out there, allowing developers to innovate in even more sectors than we had imagined. 

Our lab in miniature

Arctic Raspberries

We spent two days in Oulu this week, working with Tony & co from Symbio “rapid prototyping team”. They had done a great job in short time, and we now have a functioning hardware prototype! After testing porting OpenWrt and Zigbee USB dongle to Raspberry Pi, Pandaboard and SABRE Lite development boards, we decided to go with Raspberry Pi. It has better support from the community, and is easily available to any DIY geek that want to try our technology out at home. 

Our Raspberry Pi with Zigbee USB dongle

The compromise is that we cannot support too much of transcoding and other CPU heavy operations in the prototype. But that’s ok for prototype and lab experiments.

It was our first visit to Oulu, but definitely not the last. Really impressive competence cluster, located just short of the Arctic Circle. Given recent changes at Nokia, there are now heaps of start-ups doing world class R&D and innovation, mostly focused on smart and embedded devices. Oulu rocks!

The Interconnected Home

Just listened to a GigaOm podcast on what is wrong with the Internet of Things - Om Malik and Stacey Higginbotham discussing what makes sense and what doesn't. Om Malik talks about things not being ready yet, and that devices need to be more transparent and work with each other, introducing a couple of (to me) new phrases - the "interconnected home" and "in-home computing fabric". Well worth listening to:


Stacey is in my opinion the most insightful blogger on Smart Homes and Internet of Things, so if you want to stay up to date just follow her Twitter

Prototyping on Steroids

Domos is primarily an operating system, i.e. a software platform. This is also where we have our core competence. So when we realised we had to make hardware prototype to be able to connect all the cool devices and sensors out there, we clearly felt outside our comfort zone. I had been working towards big OEMs in China, and knew the upfront costs of getting to prototype level could easily run up to $100k. And even worse, it could take months to make. Not great for neither budget nor momentum.

We have now reduced the prototype cost by 90%. And delivery time from months to days. How? Thanks to very cool rapid prototyping technologies:

1. Low-cost hardware development boards like Raspberry Pi, Pandaboard etc have been adopted by makers and geeks around the world. The ecosystems build-in support in open source software projects. In our case, OpenWrt - a Linux distribution for embedded devices that included support for Wifi access point functionality – recently added communities working on RPi and other development boards.



Raspberry Pi - a $35 PC (image from PCMag.com)    

2. Additive manufacturing We’ll be using a technique called Selective Laser Sintering that provide robust, industrial grade casing for our prototypes at zero upfront cost. Best of all, we can test out design ideas at low cost before making final decision.
   

   

   SLS printed house

These rapid prototyping techniques almost completely remove the upfront investment. In return we get a higher unit cost of around $500 - $1,000 per unit. But we only need 10 units to power our Smart Home Labs, so for us it is a no-brainer. And we can do all this in Europe, allowing rapid and iterative development. The big takeaway is that innovators can test their creative ideas in rapid iterations with very limited upfront investment. The result? An explosion of cool, smart gadgets and wearable devices.

I just love technology.

Why we need an OS for the smart home

Homes are slowly getting smarter, but are still far away from reaching its mass market potential. We now have phones that can help us navigate the world, but does very little inside our four walls. We have cars that will soon be on auto-pilot and wearable tech that make Minority Report look yesterday. Surely our homes should be up next?

Yes, they will – but at the current rate of innovation, no time soon. There is no lack of great technology, and the value proposition should be compelling. So what is stopping adoption? The simple answer is – your wifi router.

The wifi router is the single point of connection between all cool sensors, gadgets and devices that are available for the home. They all need to connect through your wifi to connect to their respective cloud and mobile apps. And yet, there has been almost zero innovation on wifi routers when it comes to enabling the smart home. Innovation for smart homes is naturally app-centric these days. And why shouldn't it be, of course you would want to control your house through your mobile?

The answer is “yes”. And “no”. You want the apps. What you don’t want is a separate app for each device. And a separate cloud login. And a separate connector hub. And a separate protocol. There are great smart things out there today, but the underlying infrastructure is simply not scalable.

Smart home infrastructure without Domos

This is why we need the operating system. And why it needs to be on the wifi router. We need to decouple devices from apps. Let device makers make devices. Let developers make apps. Enable open competition and innovation.

In technical terms, the operating system adds an abstraction layer that enables developers to make one app that leverage all connected devices, on any protocol.

The devices are here. The apps are here. The open protocols are here. The smart router OS is the missing link. 

We plan to build it.

Back in da Hub

Another London workshop, and meeting Bill and Dorien from a cool product development company called ARRK. The prototype casing was a big concern for us, expecting large upfront costs on tooling and moulds. Not very bootstrap friendly. We have no need to worry though.

Dorien showed us some super samples of what they can do in prototyping with laser 3D printing and cuts. Exactly what we need. Cost is affordable and process very flexible. But what really impressed was the quality of the end prototypes he showed us. Got to love tech innovation.

"Seriously, you have 3D printed all this?"