An S3 Storage Experiment

My team at SUSE is working on a new S3-compatible storage solution for Kubernetes, based on Ceph’s RADOS Gateway (RGW), except without any of the RADOS bits. The idea is that you can deploy our s3gw container on top of Longhorn (which provides the underlying replicated storage), and all this is running in your Kubernetes cluster, along with your applications which thus have convenient access to a local S3-compatible object store.

We’ve done this by adding a new storage backend to RGW. The approach we’ve taken is to use SQLite for metadata, with object data stored as files in a regular filesystem. This works quite neatly in a Kubernetes cluster with Longhorn, because Longhorn can provide a persistent volume (think: an ext4 filesystem), on which s3gw can store its SQLite database and object data files. If you’d like to kick the tyres, check out Giuseppe’s deployment tutorial for the 0.2.0 release, but bear in mind that as I’m writing this we’re all the way up to 0.4.0 so some details may have changed.

While s3gw on Longhorn on Kubernetes remains our primary focus for this project, the fact that this thing only needs a filesystem for backing storage means it can be run on top of just about anything. Given “just about anything” includes an old school two node Pacemaker cluster with DRBD for replicated storage, why not give that a try? I kinda like the idea of a good solid highly available S3-compatible storage solution that you could shove into the bottom of a rack somewhere without too much difficulty.

Continue reading

Cloud Infrastructure, Distributed Storage and High Availability at LCA 2013

I’m pleased to announce that we will be holding a one day Cloud Infrastructure, Distributed Storage and High Availability mini conference on Monday 28 January 2013 as part of 2013 in Canberra, Australia.

This miniconf is about building reliable infrastructure, from two-node HA failover pairs to multi-thousand-core cloud systems. You might like to think of it as a sequel to the LCA 2012 High Availability and Distributed Storage miniconf (videos here).

Do any of the following describe you?

  • You’re building cloud infrastructure for others to use (openstack, cloudstack, eucalyptus, …)
  • Your data needs to be reliably available everywhere (ceph, glusterfs, drbd, …)
  • Your system absolutely must be up all the time (pacemaker, corosync, …)

If so, this is the miniconf for you! Please consider submitting a presentation at

We’re expecting most talk slots to be 25 minutes (including questions and changeover), but there will be openings for shorter lightning talks and maybe a couple of longer talks. CFP closes on Sunday November 4, 2012. Notifications of acceptance will be emailed out after this date.

Note that there is also an OpenStack-specific miniconf running on Tuesday 29 January. We’re hoping this will give us a pretty awesome two-day LCA 2013 CloudFest. As a rough rule of thumb, more generic or infrastructure-related talks should go to Cloud, Distributed Storage & HA, while deeper OpenStack-specific talks should probably go to the OpenStack miniconf. If in doubt, or if you have any other questions, please contact me directly.