I wanted to write this post a while ago but decided to wait until I had made my job transition. To be clear though, my thoughts have not materially changed from what it was when I was a Rackspace employee. I’ve been saying the same things for over a year, as folks from various storage vendors I’ve consulted with can attest and as John Griffith, Cinder PTL, can also confirm since he and I’ve had similar discussions over the past 7 months or so.
And of course, it was John’s blog post, The Problem with SDS under Cinder, that prompted my response, along with those of James Ruddy and Chad Sakac. In his excellent post, John expresses concern over the inclusion of any SDS software in the Cinder project because 1) he feels a Software-Defined Storage (SDS) solution would only duplicate the function of Cinder and 2) an SDS driver would encourage vendors to work on only their solutions at the expense of the greater Cinder project. Before responding to John’s concerns, let me make it clear that John and I are friends and I have the utmost respect for the work he has done as the Cinder PTL. In fact, I voted for John’s re-election and also voted for him to be an OpenStack Foundation Board member.
Answering John’s first point, James and Chad both make the case for the value of a data plane service in an SDS solution, such as EMC’s ViPR, which allows the pooling of different storage resources and applying different data services, such as block, file, and object to that storage pool. Chad also stated his agreement with John’s point on the potential redundancy of two control plane while still affirming the value of a specific SDS control plane in a hybrid infrastructure. Since I am mostly in agreement with my new EMC colleagues on these points, I will focus on two areas of disagreement I have with John regarding the perceived duplication of functions between a SDS solution, such as ViPR, and Cinder.
First, while it is true that both ViPR and Cinder abstracts physical storage arrays and provides block storage as a service to upstream clients, I would argue that this functionality alone does not defined what it means to be Software-Defined Storage. As I written in an earlier post, SDS is not strictly about the abstraction of physical storage or even the abstraction of the control plane from the data plane. It is primarily about creating an infrastructure layer that is fully programmable and able to dynamically change to match requirements of the application layer. This programmability needs to extend down into the actual storage solution itself so that attributes such as RAID configurations, QoS policies, and replication policies can be defined and programmatically changed by the application layer, via the SDS control plane. I would assert that while SDS solutions like ViPR is well on the way to getting us to that type “nirvana,” Cinder is neither currently capable of or necessarily focused on this level of integration. This is not a criticism of the Cinder project since it is not clear to me that this is the functionality that Cinder should be targeting to include. In either case, including something like ViPR into Cinder provides a clear way for customer to gain the benefits of a full SDS solution if they so desire and should we not be allowing customers to make that choice?
Second, some may argue that if storage vendors with SDS solutions want that functionality in Cinder, they should go ahead and just donate the code to the Cinder core instead of trying to get that in as a driver. But is that actually a valid response given how the OpenStack project has historically dealt with its vendor ecosystem? Instead of requiring vendors to donate ALL their code, we’ve always asked them to contribute to core functionality while allowing them to innovated and add value via integration of their proprietory solutions. Examples of this include QoS and storage replication, both of which are features that John and the rest of the Cinder Technical Committee (TC) has been working on diligently. In both cases, we’ve allowed the vendors to “plug in” their own QoS and/or replication technologies while the Cinder team, in parallel, has been working on building those features into Cinder core (By core here, I mean features that do not require a vendor driver or solution and which can be implemented in LVM, which is the base Cinder reference implementation). Would it not make sense, if there is value in having a full SDS solution in Cinder, to allow vendors to innovate there as they have done with QoS and replication, even if Cinder has a subset of SDS capabilities?
Regarding John’s second concern that vendors with an SDS controller will focus only on their products and not participate in improving the overall Cinder project, James does a good job of addressing this from an EMC perspective and making a case for why it would be in our best interest to continue supporting drivers for each of our storage backends as well as for ViPR. I would also add that I don’t see the connection between “merge[ing] not only proprietary hardware but also proprietary software in to the project” and vendors choosing to “ignore the core Cinder project.” If that was the case, then we would need to argue all the hardware vendors working on the Neutron project, and particularly those working on plugging in OpenDaylight must be guilty of ignoring the core Neutron project. But the fact is that “merging proprietory software” has little relation to a vendor’s committment to the core project, be it Cinder or Neuron. That’s proven out by the fact that the Cinder project has probably an equal number of hardware only vendors who do little more than contribute a driver to integrate their array(s) to Cinder; working only on hardware does not make those vendors any more committed to improving the core project. That commitment has much more to do with rather a vendor fully appreciates that fact that improving OpenStack core will help drive adoption of OpenStack as a cloud platform which ultimately benefits that vendor as well as everyone else.
Finally, John and others on the Cinder TC have discussed setting certain criteria that vendors with SDS solutions have to fulfill in order for their solutions to be considered for inclusion in Cinder. The criteria include contributing code to improve the Cinder core and to improve Continuous Integration testing within Cinder. I can certainly see the potential value of setting such criteria so that the entire OpenStack community is moving the project forward. However, I would implore the Cinder TC to apply those criteria fairly and evenly for all vendors if they are to be applied at all, not just those proposing a SDS solution. If as John and I have frequently discussed and he himself mentioned in an interview I conducted with him, that only about half of the vendors are contributing anything other than their own drivers, why don’t we apply the same criteria to all these vendors as well. Essentially, that would mean no vendor will have a driver approved unless they can demonstrate that they are contributing, in a tangible way, to improve Cinder core. The question then to be considered must be “is that the bar we want to set for participation in the project?”
As I’ve indicated throughout this post, I have enormous respect for John and what he and the Cinder TC has done to make Cinder a first class OpenStack project. My purpose here is not to call them out but to make an argument for how SDS can enhance the project and also to call for fair treatment of all participants. And ultimately, users and the broader community should have the final say in deciding if SDS solutions are of value to them as they look to implement OpenStack storage.
Update: John Griffith has already responded to my post with a well -reasoned follow-up post of his own. You can read that here, as well as my comment (once it has been approved). More on this to come, I am sure!
[…] up blogs from my initial posting. The most recent of which came from my good friend Kenneth Hui The problem sds under openstack cinder solves. I’m likely to say some “unpopular” things in this post and I really hope that […]
Your very well thought out technical arguments tend to a slippery slope and may well result in Cinder becoming only useful if you pay a license to someone offering their “value added” (freedom removed) pieces of hardware and software. That would be very shortsighted and silly, and *largely* independent from any technical issue. Ultimately, that’s now what OpenStack is (or what any other large, influential, long-term open source project). Cinder is part of OpenStack, an open source project whose very basic promise is to be good to use when taken straight from upstream (the first ‘open’ on https://wiki.openstack.org/wiki/Open). I think the TC will have to be very careful in implementing changes that will put OpenStack’s promises in danger. This is the first thing to keep in mind, technical issues come (slightly) after.
While I appreciate yours and John’s concerns, I believe you are creating a false dilemma scenario. SDS controllers do not replace Cinder but adds functionality to Cinder much in the same way that current storage solutions provide other added functionality to users of Cinder. There is no inherent reason why we could not continue to enhance Cinder while exposing those added-value features. For example, Cinder now has a specs field that enable a static form of QoS but still allow users to plug in storage with dynamic QoS such as SolidFire. Based on your reasoning, should we require SolidFire to either disable their QoS or to open source the code behind their QoS? We don’t do that because we want vendors to continue adding their value while also expecting that they will improve the native QoS functionality in Cinder. In the same way SDS vendors can provide added functionality while still working to improve the current capabilities of Cinder.
I see your point. As I mentioned, my main concern is to keep Cinder relevant and useful as an open source tool. All I hear about the many SDS controllers popping up is that they seem to deprive Cinder of functionality moving them lower in the stack and closer to the proprietary products they intend to ‘enable’. That’s not added value in my book, that deprivation of choice and vendor independence that OpenStack and open source in general promise to deliver.
Wasn’t Cinder designed to be such ‘SDS’ service? If it’s failing at it, do we need to rethink its strategy and implementation?
I share the same concerns you have. I am now of the opinion that we need to have a fuller discussion about what SDS actually means in the context of OpenStack storage, similar to the discussion for the Neutron project. I don’t believe Cinder provides SDS service and I am not sure if that is the intent, any more than AWS EBS could be considered SDS.
I think we should have an open discussion about this topic and decides if full SDS functionality should in fact be part of OpenStack. If it is, I think it makes sense to define what that looks like and then work as a community towards adding those services. But then as it is the case with features like QoS and replication, we should allow vendors to fill the gaps that exist and to innovate beyond the functionality provided by the reference implementation.
[…] won’t enter into technical details because it’s not my role. John Griffith and Ken Hui have a lot of interesting things to say about this and I suggest you to read their posts. My […]
[…] this promise needs to be reminded: There will be no “Enterprise Edition”. John Griffith and Ken Hui have a lot of interesting things to say about this and I suggest you to read their posts and read […]