Jay may or may not have 99 problems, but passion isn't one! Jay, like previous guests John Dickinson and Monty Taylor also started with OpenStack in the early days at Rackspace. We learned a lot about the "Big Tent" initiative in this podcast, and how OpenStack is working to balance inclusivity and innovation. (They aren't mutually exclusive.) Watch the video, download the podcast, or read the transcripts below to learn more about:

• Jay's accidental foray into tech
• The importance of quality code and asking questions
• Being realistic when it comes to definition of "production ready"
• Defaulting to "open" (and still being able to monetize)
• Emerging tech and dealing with legacy investments

You can follow Jay on Twitter at @jaypipes and on IRC If you're at the Vancouver Summit, check out his talk with Chris Dent, "Why APIs Matter."

Jeff and I are taking the show to the OpenStack Summit in Vancouver! If you'd like to be considered for a guest spot, tweet us at @openstackpod

See past episodes, subscribe, or view the upcoming schedule on the OSPod website.

To see the full transcript of this episode, click "Read more" below

Jeff Dickey:                All right good morning everyone, I'm Jeff Dickey from Redapt.

Niki Acosta:               You look dashing today Jeff.

Jeff Dickey:                Thank you.

Niki Acosta:               I'm Niki Acosta from Cisco and we have an awesome guest with us today. I am so excited to have Jay Pipes, Director of Engineering at Mirantis joining us. I've known Jay for quite a long time now. There are probably few people who've been involved in OpenStack as long as Jay have and has such a wide range of experience with OpenStack as Jay has, so welcome to the show Jay. We are so incredibly happy to have you. You can expound on your title a little more if I butchered it in any way.

Jay Pipes:                   Nah, you did just fine.

Niki Acosta:               You focused on ...

Jay Pipes:                   Nah...I work on OpenStck stuff.... how's that?

Niki Acosta:               That's great, love it. Awesome and you're even sporting your OpenStack shirt. Love that.

Jay Pipes:                   Yeah, this is the Icehouse one. Actually, this is the only summit I didn't get to go to. The Icehouse was Hong Kong, right? I did not get to go to Hong Kong. I was working for AT&T at the time and I think only one of my colleagues went. Josh Kleinpeter and Toby Forth. Yeah, this is the only one I didn't get a chance to go to so I'm actually looking forward to going to Tokyo in November and doing a summit in Asia.

Niki Acosta:               Yeah, exciting. We typically like to start the show by asking you how you got into tech and you can take that all the way up through your very colored OpenStack career.

Jay Pipes:                   Okay, well let's see. I actually went to school for Political Science which is kind of odd. Right before I went to continue my studies after my first college and went to my second college, I was working in the records retention area at UPS in Columbus, Ohio. The job was to maintain this set of bankers boxes. This is mid-90s and the accountants ... It was at the Midwest regional accounting office. This is grand stuff.

The accountants would come back from accounts receivable or payable or whatever and they would ask for a set or records for a customer and a time period. Me and this other fellow were responsible for going through this warehouse and it was very much sort of, you know, the end of Raiders of the Lost Ark? It was like that, completely disorganized, boxes everywhere, nobody took records of anything and so we would go back into the shelves, walk up the little staircase shelving unit things and find these sets of records. Very manual process. Anyway, we had this one desk in the records retention area and there was this old DOS 286 green-screen computer. There was this big book of semantic Q&A database programming. Like you look back at it now and it wasn't really database programming, it was more like draw a screen form, then put in your data or whatever. Anyway, so I took those books home over a weekend and came back and I programmed this little database application that stored all the records, all the bankers boxes and where they were, what row they were, how high and up, all that kind of stuff. That was my first foray into computing.

The accounting people got word that there's this weird guy in the back that works in the records area that created this weird program, maybe we can ask him to do some collections stuff. They brought me into the accounting office and I programmed up a little database application so that they could track their collections calls. Anyway, went from there into doing other database stuff and then I did a lot of work in Microsoft MySQL server programming. Then I switched directions, I went into Open Source. I wrote a book on the MySQL database server. Then I got hired by MySQL to run their community relations. Then went back into engineering. Did work in the Drizzle Database Kernel which was like a fork of MySQL. It was kind of like an R&D thing. One year at Sun Microsystems, then I landed at Rackspace. Moved from doing database programming to working on OpenStack when it became OpenStack.

Myself and a guy named Eric Day were one of 2 engineers that a guy named John Purrier had asked to do the due diligence about what would be the next sort of generation of the Rackspace cloud servers platform which at the time was sort of like Slicehost and it was it's own thing. They were investigating what new platform of framework they could use to build out the next generation of stuff. We had looked at a bunch of different solutions that were out there and at the time, Nova had just been open sourced and kind of like thrown over the wall by the guys at ANSO Labs. We got wind of it and we checked it out, and we're like, "Well, I mean this should work. This actually has a lot of pluses to it. We can work with this." It's sort of like in the wheelhouse of what the Rackspace cloud folks were used to as far as Python and you know, some of the technologies that were used. That's how we got involved in OpenStack, before it was OpenStack. When it was just like let's find out what the next generation of the Rackspace cloud might look like or something. That was kind of how I got into tech and all the way into OpenStack land. Condensed into whatever that was, 3, 4, 5 minutes.

Niki Acosta:               You've actually been with a few companies with OpenStack? You spent some time ...

Jay Pipes:                   Yes. What was it you said earlier?

Niki Acosta:               You're like the OpenStack record, you get around?

Jay Pipes:                   Yes, that was a great way of describing it. Thank you, Niki. I started in the OpenStack world at Rackspace because Rackspace acquired the whole team from Sun Microsystems that worked on Drizzle, the Drizzle database kernel. From Rackspace, I did a short stint at HP and then went to work at AT&T where I focused on a Dev Ops type of role. We were using a lot of Chef, we did a lot of automation GLU code. From AT&T, I left and went to Mirantis which is where I'm at now. Been here for about a year and a half or so.

Niki Acosta:               For a company its size, you know Mirantis, it always floors me to see the amount of contributions that Mirantis makes to OpenStack. Certainly understanding that they were a company kind of born out of OpenStack, right?

Jay Pipes:                   Well, I mean certainly for the last 4 or 5 years we've been focused on OpenStack. I mean, Mirantis has actually been around for awhile. A lot of the early Mirantis stuff was doing sort of offshore development work for network companies. The Ciscos of the world and that kind of stuff. We did network engineering, software engineering, foreign network devices and load balancers, that kind of thing. That was kind of the bread and butter of Mirantis before OpenStack became our sort of soul as a company. You know, since we joined OpenStack, the company has grown substantially. I think we're at some hundred and something folks now. When I joined, I think it was just shy of 600. I think we've added like a 120 or so in a year which is a decent size growth for a company our size. We're totally focused on OpenStack now. We do ...

Niki Acosta:               But the proportion of people at Mirantis actually contributing to OpenStack is still quite high, right?

Jay Pipes:                   Absolutely, yeah. I mean it's ... We have a few folks that contribute to external things on the Linux platform layer. For the most part, if you're an engineer working at Mirantis, you're working on OpenStack stuff. Whether it's foundational type stuff like Nova, Neutron, Glance, Ironic, that kind of thing or whether it's the higher level stuff like Heat and Sahara and Murano or Rally or those kind of projects that consume the other projects, yeah. If you're an engineer working at Mirantis, you're working on OpenStack, pretty much.

Niki Acosta:               Brilliant. Jeff, you have a question? I think we wanted to talk about some of the big tent stuff, right?

Jeff Dickey:                Yeah, absolutely. Jay, can you first describe kind of Big Tent and then kind of go into that?

Jay Pipes:                   Sure, I'll give it my best shot. I think there's a number of grey areas that I think people have gotten confused about with the whole Big Tent thing. Just to lay the groundwork. All of last year, literally the entire year, the technical committee debated how we define what is "core OpenStack". The whole idea of what is core was something that we debated hotly between all of the members of the technical committee and on the public mailing list as well ... Well, all the mailing lists are public. On the OpenStack dead mailing list, there were a number of conversations about like what is OpenStack supposed to be? Is it just this small group of compute and storage things and maybe some networking in there? Is it that and platform layer stuff? Do we allow competition within a specific space? For instance the deployment space? Should we be welcoming of Puppet and Chef and Ansible and SaltStack and CFEngine and TripleO ... It's sort of way of doing deployments. There were back and forth debates on what OpenStack should be and a number of us were pushing the idea of being a Big Tent. Being very inclusive of newcomers and also allowing competition within specific spaces as well as focusing on the API's, the public restful API's that define that interface between the client and the server.

At the end of the year, we had this sort of giant project structure reform proposal that Terry and Doug Hellman and a number of other folks had crafted to sort of represent the distillation of all the debates that we've been having over the year. It boiled down to a number of major changes. The first one was that we got rid of the idea of programs being, like there was a compute program and a deployment program and a network program, right? Let's take an example of the deployment program. The deployment program was TripleO. One of the things that we said was if we say the deployment program for OpenStack is TripleO, then by the nature of saying that we're kind of blessing that as the TC, then that's the way to deploy OpenStack. We are on the same token saying, "Well then, maybe it's not appropriate to deploy OpenStack with Chef or with Puppet or Ansible or SaltStack." That's not the real world, right? In the real world, people are deploying OpenStack with Chef and Puppet and Ansible. We were saying, "Well, let's get rid of these broad categories like the deployment program and have it be blessed by the TC and instead just let other things into the OpenStack ecosystem and not necessarily bless them, but say you are as much a part of the OpenStack ecosystem as this thing called TripleO." That was the first big thing, is breaking down these sort of silos that we had to put everything in and then have the TC bless whatever was in that silo as being OpenStack. That was the first thing.

The second thing was being much more objective about how to get into the OpenStack code namespace. Prior to the Big Tent initiative and the reform of the project structure, we had this thing called incubation. A project would apply to be incubated and over a period of time, the project would come before the technical committee and the technical committee would go through an incubation review and then a series of graduation reviews where you would graduate from incubation into the integrated release. Then, I guess supposedly if you get into the integrated release then you are part of core OpenStack and the technical committee blesses these set of projects as being production ready and scalable and mature and all these stuff.

Well, the problem with that is that these incubation and graduation reviews partly by nature of the technical committee being elected in 6 month cycles and yearly cycles, would get really inconsistent feedback in these reviews. Some of the feedback was entirely subjective. If you take an example of Zaqar which is the message queue as a service or the queuing service but tenant facing. We had a number of reviews, I think it was 3 graduation reviews for Zaqar, where they were incubated and they were trying to get into the integrated release and go through the graduation review with the technical committee. We had folks on the technical committee bringing up some fairly valid points, but were pretty subjective and even inconsistent from the last time that Zaqar had gone through their graduation review. We were telling them sort of like different things from one graduation review to the next based on some of the TC member's ideas about appropriate architecture or appropriate infrastructure or appropriate dependencies that the project might have.

In the end, it just kind of stifled innovation. It was preventing a good project contributor team from making progress because we were sort of almost throwing arbitrary restrictions at them. The second big part of the Big Tent initiative was to make the set of requirements for going into the OpenStack code namespace be as objective as possible. Now, the list of things that you need to do to get into the OpenStack code namespace is you need to follow the 4 opens of OpenStack, so Open Design, Open Collaboration, Open Source and Open Reviews. Those 4 things are very objective. You can go and see whether or not you're doing open code reviews, whether you're discussing design in the open and all that kind of stuff. You also need to say that if you're in the OpenStack code namespace that you will defer to the technical committee for any reasons where you can't reach an agreement within your project contributor team. There's one other objective requirement that I'm forgetting, but the point being, it's not this subjective, arbitrary set of requirements that individual TC members might come up with about, "Well, the architecture isn't as scalable or as distributed as we'd like and therefore we are going to deny you access into the OpenStack code namespace." Those were the first 2 big things, which is opening up the OpenStack code namespace to more applicants and being more objective about the requirements for getting into the OpenStack code namespace.

Finally, the third big thing of the project structure reform called Big Tent is instead of having this one binary integrated release bit of information that we apply to projects in the OpenStack sphere. Instead, break that into very fine grain tags. Instead of just you're in the integrated release and that somehow would mean that Celiometer or Neutron or Nova and all the projects that were in the integrated release are of the same quality, same code maturities, some production readiness, it doesn't make any sense. To say that Swift, the code base of Swift and the code base of Ceilometer were at the same point in their maturity or the release cycle is ludicrous. In the real world, that's just not the case and so we wanted to come up with a set of tags that were much more fine grained and informative to our audience, whether the audience was a developer, whether the audience was a deployer, a packager of OpenStack or an operator. We are in the process of coming up with these set of tags, that hopefully will indicate to these 4 different audiences, more descriptive information other than just you're in or out of the integrated release. We've come up with a set of tags that describe the release process whether or not it's managed, what cycle it's on. We've come up with a tag that describes the diversity of the contributor community whether or not it's entirely dominated by a single company or whether it's spread out amongst multiple companies. You know, that's an indication of whether or not the road map for a particular project is going to be driven by a product management within one company alone versus a road map that's drive more by compromising consensus. These tags are not meant to be blessed or not blessed. They're meant to be very specific pieces of information that we give to users to let them make a decision. If they don't want to use any projects that are predominantly or entirely driven by one company, well now they have a tag that will give them that information. That's the third piece of the Big Tent initiative that, I think, was an important component of it.

Niki Acosta:               The net of this is that anyone who goes and chooses to participate in OpenStack will have some guard rails it sounds like, but probably from what I can tell, I think the benefit really is going to be to people who are actually consuming OpenStack, right? To be able to not make the decision for them but to give them the information they need to make the best decision based on their use case or their application or how they want to operate it or whatever, right?

Jay Pipes:                   It's kind of always been my opinion that having these 13 individuals on the technical committee pronounced whether or not they think something is production quality or whether they think something is scalable is just kind of ridiculous. I mean, first of all, the 13 individuals on the technical committee have a very wide range of experience in operating OpenStack. Some have no experience whatsoever, others have a little bit, some have quite a bit. To have that set of individuals pronounce project X is production ready, I don't think that's all that useful and in fact, having them make that stance and bless something as production ready, I think actually did a disservice to our users because you're giving out false information or misinformation.

Instead, I prefer to say have the operators provide feedback on what they think is passed and experimental phase or that they have experience in running at scale, in a stable way for X number of months or release cycles or whatever it so that you give the operator community and the deployer community and the packaging community more information than just, "Oh, it's in the integrated release." I know there's a lot of people that want that sort of check box like, "Oh, it's in the OpenStack integrated release." I actually made this case about this to a customer of Mirantis recently, you know, would you rather have misinformation or sort of pretend information just so that you feel good about some project being in the integrated release or would you rather have information that's coming directly from operator and deployer feedback that's saying, "Maybe this needs another 3 to 6 months to bake."

That's where I stand on that. I think that the distributions of OpenStack where there's Mirantis or RDO or a bunch of cluster archive and all these other things, those distributions will make a value judgment on the projects based on what they are able to test, functionally and at scale in their labs. I think that should be the indication to the operator community that they should take to hear, not something that the technical community has proclaimed.

Niki Acosta:               In all of these, I mean, there's obviously a point to be made about community contribution and the process of growing core teams. How do you flush that out in your mind?

Jay Pipes:                   That is like the perennial question that any of us that are on the core team, for any of the sort of busy or high velocity projects, we struggle with all the time. I mean, we've gone through all sorts of different proposals, like how do we merge more codes but at the same time keep the quality high? How do we get more individuals into the core teams so that we can spread the load, the review load, out? Nova, we struggled with this mightily. We've lost a lot of core team members because they move on to other things. There are more important things that their companies assigned them to work on and that's completely natural, but if you've got 10 individuals that are responsible for [inaudible 00:24:05] a piece of code which allows it to go through gate and get merged into the code trees. For a project like Nova, we typically have anywhere between 550 and 650 active patches in review.

I mean, for 10 people to take that weight, it's just impossible. What ends up happening is that the core team members sort of just get into this, I don't know, flow of prioritizing stuff in their head and a lot of times, you'll get core team members that simply have to ignore large chunks of the code base because they have to focus on one particular area. Otherwise, they'll never get anything done. They become sort of lieutenants of a particular coding area where other core members trust them more than other areas of the code. I hate to say it but the squeaky wheel, it gets the grease a lot of times and a lot of people come to me and say, "Why does it take 3, 6 months to get my code merged?" and I'm like, "Well, did you come onto IRC and asked anyone to do code reviews?" "No." A lot of it is like as simple as just hop on IRC and see if you can get one of the core team member's attention and just get someone to look at it.

Niki Acosta:               What are you looking for in a good code review? Can any part of this be automated in some way?

Jay Pipes:                   A lot of the stuff is automated when it comes to style and white space checks and what we call [PEP 8 checks 00:25:51]in Python land, but there are ... There's not a whole lot more that we could automate in my opinion. There are some things around like code coverage, like unit test code coverage percentage that we might be able to automate somewhat but not entirely. It's not something I would support as a gating measure that would like download your patch and keep it out of the gate.

There are a few things that when I'm trying to sort of mentor newer contributors in the OpenStack space to how to do a code review and also how to respond to code reviews. It's not just like how do you provide useful and insightful feedback to others, but it's also how do you take that feedback that core reviewers and non-core reviewers are giving you in a way that's constructive.

Niki Acosta:               In a way, it might be like, "Hey, your baby is ugly," and someone is [crosstalk 00:26:56].

Jay Pipes:                   Exactly, everyone always thinks that. My baby is ugly, you scream, run away and your patch gets abandoned. There's a lot of that. I think I tweeted, I don't know maybe a couple of years ago now, but it cost nothing to be kind and courteous in a code review. It really is true. You see many times, reviewers will get frustrated and I completely understand that. You're overworked from the review queue. Any core reviewer on any of these high velocity projects will get requests 10 times a day or more if you're on IRC to do code reviews. It can get very frustrating but if you just take a deep breath and just say, "You know what? All this person is trying to do is just get some feedback on their code or understand something that you might not have expressed clearly enough and they're not trying to bother you." The last thing anyone is trying to do is bother you. Like if they didn't have to come on IRC and ask to get a review on something, they wouldn't. It's not like the first thing someone who wakes up in the morning and goes, "You know what? I'm going to go on IRC and bug some core reviewers today." It's not like that. You got to keep that in mind.

You also have to remember that nobody knows the answer to everything. People that sort of pretend that they know the answer to all questions, like I avoid them at all costs because you know that it's just not realistic. I always say to new contributors, ask questions on reviews. That's the number 1 thing that tells other reviewers and other patch submitters, you're interested in learning how to make the code better, you're interested in the code that you submitted a patch to and you honestly want to learn, so if you don't understand something, ask the question. Ask good questions. Also, I often tell people that if you have a question about how a particular piece of code in a patch works, it's typically an indication that that piece of code could use a good doc string or a good code comment explaining why the patch submitter is doing what they're doing.

These seemed like really small and like petty things but if you think about it, a code comment ... Code is read, what? A hundred times, a thousand times more often than it's written. If you can write a good code comment or if you're going to ask a question that will prompt a code comment being introduced into the code that will benefit a thousand people that are reading that code after you, then you've done a really great review. If you think about it, you're not doing the review saying, "You should do this, you should do that, you should do this." If you're asking a question and that question results in a good code comment that lives on, that explains to people why that code is doing what it's doing, you've effected a hugely important thing in the code base just by asking a question. That's my biggest advice to new people is ask questions on reviews. The +1, -1 stuff, you know, I don't like to encourage people to be nit picky about style or stuff like that. I can say, "You know what? Next time, do this. If you re-spin this, maybe change this," but it's the questions that you ask and how you ask them and the curiosity that you show, that really makes a difference for new reviewers. That would be my best piece of advice to people that want to do more code reviews.

Jeff Dickey:                Jay, talking about the Big Tent and the code review stuff. Do you think OpenStack is too open?

Jay Pipes:                   Great question, Jeff. I don't. I'd like to think that everything that I personally do, both in my community work as well as Mirantis, I default to open. This is something that ... Actually, when I use to work MySQL, Marten Mickos, the CEO at MySQL at the time who then went to Eucalyptus and then HP. He used to say this is default to open. If there's not a reason ... There should be a reason to keep something from being open, not the opposite of that. In my mind, I'd rather have more information, more open and transparent discussions even if it is chaotic, even if it is sometimes very difficult to reach consensus and to get decorum on something. I'd rather it'd be that way than have 2 people fight it out in a room and then let a bunch of other people go do some work and then you throw it over a wall.

I see that too many times both internally at Mirantis, you know, there are some teams that have a tendency to do that because they come from a proprietary software development world. There's a number of companies in the OpenStack business that would do the same thing. I try and discourage that as much as possible because I do think that long term, the benefits of everything being open and on the table and transparent are ... They outweigh the cons of the chaos that sometimes ensues.

Niki Acosta:               Love what you're saying. If you're defaulting to open, how do you make money? Like how do you make money in OpenStack?

Jay Pipes:                   Well, there's ... Just because you're defaulting to open and having an open road map and an open conversation about your designs and your implementation, doesn't mean that you can't charge for things. For instance, the Mirantis OpenStack distribution just ... RDO and any other company that's attempting to productize a distribution of OpenStack, there's nothing preventing them from charging subscription rates for the work that is being done by those companies. That doesn't mean that the road map for the products and the projects that are being worked on, that make up that distribution in any way, needs to be closed. I would turn the question on its head and say what prevents companies that have an open road map for making money? I don't think anything does.

Niki Acosta:               That's a good point. I think it [crosstalk 00:33:50].

Jay Pipes:                   It strengthens the product to be able to say to any customer or competitor, "Hey, is your road map completely transparent