An S3 Storage Experiment

My team at SUSE is working on a new S3-compatible storage solution for Kubernetes, based on Ceph’s RADOS Gateway (RGW), except without any of the RADOS bits. The idea is that you can deploy our s3gw container on top of Longhorn (which provides the underlying replicated storage), and all this is running in your Kubernetes cluster, along with your applications which thus have convenient access to a local S3-compatible object store.

We’ve done this by adding a new storage backend to RGW. The approach we’ve taken is to use SQLite for metadata, with object data stored as files in a regular filesystem. This works quite neatly in a Kubernetes cluster with Longhorn, because Longhorn can provide a persistent volume (think: an ext4 filesystem), on which s3gw can store its SQLite database and object data files. If you’d like to kick the tyres, check out Giuseppe’s deployment tutorial for the 0.2.0 release, but bear in mind that as I’m writing this we’re all the way up to 0.4.0 so some details may have changed.

While s3gw on Longhorn on Kubernetes remains our primary focus for this project, the fact that this thing only needs a filesystem for backing storage means it can be run on top of just about anything. Given “just about anything” includes an old school two node Pacemaker cluster with DRBD for replicated storage, why not give that a try? I kinda like the idea of a good solid highly available S3-compatible storage solution that you could shove into the bottom of a rack somewhere without too much difficulty.

Continue reading

Cloud Infrastructure, Distributed Storage and High Availability at LCA 2013

I’m pleased to announce that we will be holding a one day Cloud Infrastructure, Distributed Storage and High Availability mini conference on Monday 28 January 2013 as part of 2013 in Canberra, Australia.

This miniconf is about building reliable infrastructure, from two-node HA failover pairs to multi-thousand-core cloud systems. You might like to think of it as a sequel to the LCA 2012 High Availability and Distributed Storage miniconf (videos here).

Do any of the following describe you?

  • You’re building cloud infrastructure for others to use (openstack, cloudstack, eucalyptus, …)
  • Your data needs to be reliably available everywhere (ceph, glusterfs, drbd, …)
  • Your system absolutely must be up all the time (pacemaker, corosync, …)

If so, this is the miniconf for you! Please consider submitting a presentation at

We’re expecting most talk slots to be 25 minutes (including questions and changeover), but there will be openings for shorter lightning talks and maybe a couple of longer talks. CFP closes on Sunday November 4, 2012. Notifications of acceptance will be emailed out after this date.

Note that there is also an OpenStack-specific miniconf running on Tuesday 29 January. We’re hoping this will give us a pretty awesome two-day LCA 2013 CloudFest. As a rough rule of thumb, more generic or infrastructure-related talks should go to Cloud, Distributed Storage & HA, while deeper OpenStack-specific talks should probably go to the OpenStack miniconf. If in doubt, or if you have any other questions, please contact me directly.


Shameless High Availability Googlebait

I’m sure newcomers to high availability on Linux are still being bewildered by reams of readily googlable semi-ancient information floating out there in the ether.  So I’m going to try to help remedy this by saying:

This has been a public service announcement.  Thank you for reading.

Prior Art

I’ve just been reminded that I never posted the drawing I did during the presentation that Florian and I gave at 2011.

This PDF is the one I drew during the talk, the only change being the addition of a copyright notice.  The “slides” will actually make sense if you were either present at the talk, or if you’ve seen the video.

This PDF is a slightly more polished version, with labels on each panel, which may be more appropriate for a printout and/or may make slightly more sense without the exposition present in the talk.

The drawing was done in Inkscape using a Wacom Intuos 4 tablet, so the originals are SVGs, but I’m operating under the assumption that PDF is still a more widely viewable format.  Also, the WordPress upload widget is whining about SVG files being a security risk, and I don’t want to have to convince it otherwise right now.  If anyone actually wants the SVGs, please let me know and I’ll sort something out.

One last thing before I forget: both our email addresses have changed since 2011, so don’t believe the drawing on that count.  It’s not lying about anything else though 🙂