So, it’s been a fun couple of weeks. I’ve played with mesos/marathon for quite a bit trying to build something that was simple, easy to deploy from development, highly available, and scalable. What I started to realize while putting together Ceph… I’m building Deis.

After talking to the guys working on the Deis project, I learned that my issues with brittleness had more to do with crappy linux Vm’s on Azure and less to do with the actual project’s quality. They worked with Microsoft to get it fixed and I’ve been running a nice little successful cluster for a few days now.

Needless to say, I’ve learned quite a bit about building these clusters … and everything is kinda suckish. Deis hides it all with their dedicated guys contributing bootstrapping scripts for various providers like Azure, AWS, and Digital Ocean. But at the end of the day – it just sucks. In some ways, it reminds me of the days when people actually had to know 3D math to make games, you had to build the tooling yourself and God Forbid initializing DirectX… Now, it’s what? One or two lines of code and more tools than you know what to do with?

I feel like bootstrapping the servers with the framework shouldn’t require knowing the network topology beforehand. I know, some of you are out there shaking your heads. But you remember the days when you had to know the size of your arrays beforehand too (and still do if you’re in C land). We need to figure out someway somehow to discover network topology and bootstrap the cluster from there.

I’ve been toying with the idea of using something like Serf to find other nodes in the cluster. It only needs to find one node, and since the cluster should exist in the private address space and DHCP tends to give out a nice ordered numbers… it shouldn’t be more than a few guesses to find another node. After a few seconds we can be pretty confident that we’ve found all our neighbors and can then query the serf network for instructions on how to build ourselves. A response can come back, most likely with a bootstrap script that pulls in build dependencies. In the case of starting with Consul (or zookeeper), that needs masters, the bootstrap script could run a probability that it will even provision a Consul server. If we have 100 nodes and there’s a 10% chance a node will get provisioned with Consul server, there’s a pretty good chance we’d have around 10. If our bootstrap script detects that a quorum couldn’t be formed, then the lowest ip would provision another. Repeat until quorum is formed.

That may be over complicating things a bit, but I think it is a good start.

Now, time to go back to seeing if these provisioning scripts work in Azure… I’ll let you know how it goes.