The network is ALWAYS hostile

The 4th fallacy of distributed computing is that the network is secured. It is a fallacy because sooner or later, you’ll realize that the network isn’t secured.

Case in point, Microsoft managed to put 250 million support tickets on the public internet. The underlying issue is actually pretty simple. Microsoft had five Elastic Search instances with no security or authentication.

From the emails that were sent, it seems that they were intend to be secured by separating them from the external networks using firewall rules. A configuration error meant that the firewall rule was no long applicable and they were exposed to the public internet. In this case, at least, I can give better marks than “did you really put a publicly addressable database on the internet in the days of Shodan?”

It isn’t a matter of if you’ll be attacked, it is an issue of when. And according to recent reports, the time it takes from being network accessible to being attacked is under a minute. At worst, it took less than a couple of hours for attacks to start.  If it is accessible, it will be attacked.

So it is was good from Microsoft to make sure that it wasn’t accessible, right? Except that it then became accessible. How much are you willing to bet that there was no monitoring on “these machine is not accessible from the internet”? For that matter, I’m not sure how you can write a monitoring system that check for this. The security assumptions changed, and the systems wasn’t robust to handle that. What is worse, it didn’t fail close. It failed wide open.

The underlying cause of this mess is that the assumption that you can trust the network. It is closed, secured and safe. So there was no additional line of defense.

When we designed RavenDB’s security, we started from the assumption that any RavenDB node is publicly accessible and will be attacked. As such, we don’t allow you to run RavenDB on anything but the loopback device without setting up security. Even when you are running inside locked network, you’ll still have mutual authentication between client and server, you’ll still have all communications between client and server encrypted.

Defense in depth is the only thing that make sense. Yes, it is belt and suspenders, but it means that if you have a failure, your privates aren’t hanging in the wind, waiting to be sold on the Dark Web.

When designing a system that listen to the network, you have to start from assuming you’ll be attacked. And any additional steps to reduce the attack surface are just that. They’ll reduce it, not eliminate it. Because a firewall may fail or be misconfigured, and it may not happen to you. But if a completely separate machine on your closed network has been compromised, you best hope that it won’t be able to be a bridgehead for the rest of your system.

This attack expose 250,000,000 support records(!) and it was observed because it was obvious. This is the equivalent of a big pile of money landing at your feet. It gets noticed. But let’s assume that the elastic node was an empty one, so it wouldn’t be interesting. It takes very little from having access to an unsecured server to being able to execute code on it. And then you have a bridgehead. You can then access other servers, which may be accessible from the opened server, but not for the whole wide world. If they aren’t secured, well, it doesn’t matter what your firewall rules say anymore…

The network is always hostile. You can’t assume who is on the other side, or that you aren’t being eavesdropped on. Luckily, we have fairly routine counter measures. Use TLS for everything and make sure that you authenticate. How you do it doesn’t matter that much, to be honest. User / pass over HTTPS or X509 certificate are just different options. And while I can debate which ones are the best, anything is going to better than nothing at all. This applies for in house software as well. You microservices should authenticate, even if they are running in the isolated backend.

Yes, it can be a PITA to configure and deploy, but it isn’t really something that you can give up on. Because the network is always hostile.