Symantec's Director of Cloud Platform Engineering, Yuriy Brodskiy, was a really interesting interview-not only because he was a very early adopter of OpenStack in his PayPal days, but also because he now works for one of the pioneers in software security. He gave us some surprising insights into how his company views open source in general and OpenStack in particular, as well as what they're doing to make the cloud more secure. He also discussed:

• How cloud changes the culture of an organization
• How OpenStack changed perceptions of open source software
• The upside (and downside) of rolling your own distro
• The future of PaaS and containers
• What Symantec is doing with OpenStack
• What to consider in order to create a truly secure OpenStack environment

To see who we're interviewing next, or to sign-up for the OpenStack Podcast, check out the show schedule! Interested in participating? Tweet us at @nextcast and @nikiacosta.

For a full transcript of the  interview, click read more below.

Jeff:                 All right, good morning everyone. I am Jeff Dickey from Redapt.

Niki:                I am Niki Acosta from Cisco and we have a really awesome guest with us today. Yuriy, introduce yourself.

Yuriy:              Hi, everyone. I'm Yuriy Brodskiy. I am the director of engineering here at Symantec.

Niki:                Yuriy, we're really excited to talk to you today. It sounds like you guys are doing some really cool stuff with OpenStack, but we typically start the show by asking you how you got into tech.

Yuriy:              Sure. Not a very exciting story, but start my [inaudible 00:00:41] year in QEA and spent a couple of years in different companies doing quality engineering and, ironically, now, I ended up at Symantec many years ago where I moved into development and a couple of years there on a website and I got a call from a friend of mine to join a small company called PayPal. I moved to PayPal where I spent a lot of years. Over 11 years at PayPal wearing many different hats.

During PayPal, to actually do quality automation there and build some processes, and through all those years I move all the way from QEA and somehow I ended up in Ops, and then what you have, but the last seven years of my PayPal career I was wearing different hats in operations from doing L-3 support to engineering. Here I am, the second round at Symantec, now.

Niki:                In those early days, too, I remember just as cloud was starting to hit critical mass. Prior to that, Ebay and PayPal were kind of leading the charge in terms of cloud. They were an early explorer of OpenStack most certainly. What was that like adopting cloud in an organization before cloud had really hit critical mass?

Yuriy:              It is an interesting journey there so Ebay was earlier adopter. Right? Ebay built before there was an OpenStack. Ebay built their own cloud platform, and that adopted OpenStack, actually before PayPal did, but what really on the PayPal side we knew we need it, but I don't think we actually answered how to. Really, until [Surram Mandear 00:02:47] some of you might know joined PayPal. He joined us from Ebay. He came in and started working for Surram and he said, basically, "Look. We're going to move PayPal in the cloud and we're going to use OpenStack. My first reaction was go to Google and figure out what he is talking about.

What is OpenStack? Right? Nobody really knew at PayPal what OpenStack is how to best solve our cloud vision. That was just when ESX came out and, now, looking back, it's kind of funny, but, back then, when we looked at ESX we said, "Look. It's pretty stable. It's a good platform. Let's go. Let's go build it." The story is there. It was an innocent adoption because I am pretty sure like with any company moving traditional IT processes into the cloud world it's not an easy journey. There is a technology aspect. There is a people transformation aspect, but I'm sure some of you heard PayPal's story.

I mean, PayPal had been fairly successful in that journey. Probably, but being a little bit more aggressive than some other companies by going directly into production versus starting with labs and POCs and QE environments. We are taking a very similar approach actually here at Symantec just saying, "Look. We know what OpenStack can and cannot do." We'll solve the real use cases in prod, and then when the development requirements come in, we'll solve them as well.

Niki:                How did you guys deal with finding the right people to be able to implement and run cloud at that time? I can't imagine there were a ton of people out in the market. You were like, "Yeah. I'm a DOT engineer or I'm a OpenStack Python developer or engineer." Did you guys just learn along the way?

Yuriy:              We have. Part of what, I mean, we had a bit to jumpstart. We had a big help from Mirantis. Mirantis came in and they helped us to jumpstart the effort, but to your point, there was no people who actually knew what OpenStack is. There was no cloud people. There was no devops people. If I look at our original team when we started building a cloud team, it was me who had to Google what OpenStack is. Our first engineer who is now our [inaudible 00:05:33] here at Symantec right now. He is our normal guy. He was a Hadoop engineer. Right? We hired a Hadoop guy who just was a very good developer.

That's how we really built the team, not by let's go hire OpenStack guys, but really just hiring lots of smart engineers who wants to go solve real problems.

Niki:                How are you doing with that now? I mean, obviously, I can imagine but like every other company on the face of the earth. There is still a pretty strong desire to bring in cloud talent. How are you managing that now at Symantec?

Yuriy:              It's the same, Niki. Maybe it's actually harder because back then it was people were not jumping the OpenStack bandwagon so it was much easier. Now, you're competing and especially here in the Bay competition is huge. It is very tough. It's very tough to find good talent, but, honestly, we're taking the same approach. If we find people who know OpenStack great, but really just finding great engineers who want to solve the problems. They're learning the way.

Jeff:                 Yeah. That's good. Hire smart and they'll figure it out.

Yuriy:              Pretty much, unless somebody else has some other magic. If there is, I'd love to hear it, but I haven't figured that one out yet.

Jeff:                 What were the drivers behind the decision to go with OpenStack early on in the ESX days, and at PayPal?

Yuriy:              Back then, if you look at PayPal infrastructure, as I said, we were physical world, paper and digital. We did have some virtualization on VMware and we've done all the MSCs. Right? KVM and bunch of short scripts, writing scripts. It's all good, but we knew we needed something else. The reason really wasn't let's fix it for Ops. The reason was let's fix it for dev in a sense of let's fix for agility. Right? They need to move fast.

For the whole cloud effort, what, actually my job I was running the team responsible force solution engineer, which, basically, the guys would work with developers, figure out what the needs are, and then go build it up on the production side. Right? Build the servers. Build the network. Build the [inaudible 00:08:09] etcetera. It takes time. You can imagine the business world. The world is changing and quite a bit of demand of getting our developer a bit more agile and be able to release. Everybody was looking at Googles and Facebooks of the world or even the older fancy blogs saying, "Hey, they can kick butt. It's so fast, so cool."

Of course, your executive team looks at you the same way so what are you going to do? Right? How are you going to solve it for me? Realistically, even though, yes, it was ESX, but when we looked at the option the community behind OpenStack and where OpenStack was going it was quite our best choice of just rally behind it.

Jeff:                 Yeah because I think there was a lot of choices back then. I mean, you had CloudStack and Eucalyptus and you had OpenStack.

Yuriy:              Yeah.

Jeff:                 Do you guys look at them and make the decision or was it an easy-

Yuriy:              We [inaudible 00:09:13] we have not. Right? I mean, we knew what they are, but we just basically went heads down and just said, "OpenStack it is, and we'll make it work."

Niki:                Yeah. At that time, I'm not sure that some of the other options were open source? Right? I mean, at that time I think they were still proprietary solutions?

Yuriy:              To a certain extent; I mean, some of the principles that we adopted is no vendor logins. That's what OpenStack provides. It's open API. It's you are free on your hardware and it's huge. It's a huge benefit. I mean, again, if we're 24 percent on development side it's one angle, but you do have to, at some point, start looking at the threshold costs and having the folks [inaudible 00:10:03] operational side is huge as well.

Jeff:                 You think from-

Niki:                Yeah, totally.

Jeff:                 -working with your doing OpenStack at Symantec now and you started pretty early on, do you think OpenStack is now easier or more complex from when you started?

Yuriy:              Good question. If I would take ... It depends on the angle you look at. Right? If I would take OpenStack as it is...say Icehouse or Juno, and go back into ESX days, I would say, "Hey, yes. It's much easier." It actually works, but we're not then. We're here and requirements are different because there is expectations. There is a level set and certain things that were a wild factor a couple of years ago are ... It's a minimal level of expectations today. You take that and you increase complexity by requirements.

People no longer interested in spinning up VMs. It should work. Give me something else. That's where complexity really comes in. It's stitching it all together and solving the real use cases, not just virtualization use cases.

Jeff:                 Yeah. I like that.

Niki:                You have your core projects. You've got whatever, Nova. You've got Cinder, Glance, Horizon, whatever. Have you implemented any of the ancillary projects or are you exploring any of those projects especially like Ironic or Sahara or Savanna? I forget what it's called now.

Yuriy:              Yeah. We are, Niki. The core is there, of course, you've got to run. We are, in answer to Jeff's question is it harder or easier, and that's why it's harder because the requirements are bigger so, for example, at Symantec we're looking at [inaudible 00:12:04]. We are Symantec. We are security. As you can imagine, this is near and dear to our hearts. Then we're looking at pride and working on pride such as [inaudible 00:12:15]. Symantec has been very involved with [inaudible 00:12:19] effort as well.

Ironic is definitely in our roadmap for probably we're going to start looking at it very close in the next couple of months for the same because, now, all those capabilities comes in. What it does allow us to start removing our custom and homegrown scripts. Right? Ironic, specifically. Talking about bare metal provisioning where we had to build our own system to do hardware and bare metal management. The hope is that we can solve it now with OpenStack standard APIs and start clawing away our own craft.

Niki:                Are you at liberty to discuss what your hardware stack looks like and, secondarily, does the hardware matter?

Yuriy:              I probably wouldn't go into too many details, but, I mean, there is no secret. It's generic hardware. There is nothing special about it. Data [inaudible 00:13:18]. Right? Basic to use we actually are looking at more High Def solutions right now. [Inaudible 00:13:28] performance, and, again, it's all driving project requirements, the business needs so we have a big ask on performance so we're looking at this is these solutions to be part of the platform, but, realistically. From an OpenStack perspective, we don't see it as it matters, really, what hardware you run.

There are certain fields that we're interested in terms of security from a Symantec perspective, but besides that, hardware is hardware. Right? What really matters, what we're looking at from our side because of the scale, really, dollars matter because putting the older piece of equipment in a cloud has become expensive because you're not going to be able to get the same compute power out of they start to break as if you can replace it with something newer so when you are running a lab this probably doesn't even matter.

You've got a couple of developers there spinning up VMs, but you start going at a quite larger scale and different use cases. For us, it's really important to actually have top performing equipment, part of the cloud.

Niki:                As far as OpenStack goes, it sounds like you guys have gone the DIY route, maybe, but are you guys relying on any commercial distributions of OpenStack or are you guys rolling your own?

Yuriy:              We are rolling our own. We are not. We're taking the same approach as we did at PayPal. It's basically no logins. Part of our goals here is self sufficiency so we are not relying on any distribution for that.

Niki:                If the classic enterprise VP came to you, VP, a cloud or whatever, and said, "Hey, Yuriy, we're thinking about implementing OpenStack." Would you tell them to roll their own or would you tell them now to look at commercial distributions knowing what you know about how long it's taken you to get where you are now?

Yuriy:              I would probably say it depends on the use case. If the use case is ... I mean, I wouldn't call it simple, but it's not very complex. We're trying to solve for that. We're trying to solve ... It's a chicken and the egg, Niki. Right? I mean, the reason we're building our platform, our team here, because we are here for the long run. I do not believe that in the long run you can afford because, I mean, OK. Fine. OpenStack is open source and that's one of the biggest drivers, but if you're basically going back and logging yourself in at the vendor, what's fundamentally different? That's my take. I'm not saying if I'm right or wrong, but it might be those.

Jeff:                 Yeah. I don't know if you guys saw the article that Randy Bias did recently talking about-

Yuriy:              Yeah.

Jeff:                 -the distros. I think that was interesting, but, yeah. It is one take and it does matter in your requirements and what level of support. What do you do or do you have any war stories or recommendations around upgrading because you've probably had to upgrade several times?

Yuriy:              Yep

Jeff:                 What does that look like?

Yuriy:              I have war stories and I have success stories so we got both. Right? There are definitely war stories. Actually, here we've done pretty well at Symantec. At PayPal, again, we were early days. We were learning. We didn't have enough experience on that. I'm sure the team is doing much better nowadays. It was painful. It was painful because you run your website. You can't afford taking any outages because then it impacts that's the bottom line. Right? Business doesn't really care if you're in an OpenStack meeting. We are a physical first structure. Uptime is uptime.

Not without failures, right? We had that, but, again, it was early days. I'm not going to say. I mean, there is big discussions going on right now on the forms and [inaudible 00:17:51] this or how are we going to do zero downtime upgrades. It's very important, especially for enterprises. We just actually finished our upgrade here at Symantec so here at Symantec, basically, one of the rules ... Well, I don't know if it's a rule or policy. Whatever you want to call it; guiding principle, let's put it this way. We have these when we're staying maximal one release behind.

We are on Icehouse right now. Once we come back from Vancouver we're going to get in Juno, which means we upgrade every six months. That is an expensive exercise, very expensive I think. Yeah?

Niki:                Are you at liberty to say how many people you have working on this?

Yuriy:              Sure, no problem. On the upgrade side, really, we're talking about ... Let me attack that differently. The way we have a process is the engineering team, the guys who work in the OpenStack or they are the ones who basically go in and preparing all the upgrade scripts with all the testing, but then our Neutron guys are the ones who actually execute. On the infrastructure side that's actually is going to do all the project changes and configurations and actually all the collaborating side. I think it's four to five guys. We just done our last upgrade.

I think they submitted a document for one [inaudible 00:19:28]. We upgrade our site from Havana into Icehouse with zero downtown on the data plane, and I think it was just under 10 minutes on a controlled plane of OpenStack.

Niki:                That's incredible.

Yuriy:              It is. A lot of planning, but at the end of day, it was very successful. You go from bad to better. I guess you learn. You learn what works, what doesn't, and we are where we are. Now, we're shooting for getting to five minutes once we go into Juno.

Niki:                One of the things that we talked about prior to the start of the podcast was how cloud is actually changing the culture and processes of how stuff gets done in some of these organizations that have adopted cloud. It'd be really useful to get your take on that as someone who's done it, actually.

Yuriy:              No, absolutely. It's very interesting, Niki, to see. At Symantec, it's a very new team. The cloud platform engineering team is a brand-new team at Symantec. It's what we call a startup within and, sometimes, it's an overloaded term and that was the sales pitch that was given to me when I joined Symantec. "Hey, why don't you start up with a very well financed company?" Fine. I hear that spiel all the time, but reality is it's actually somewhat true, and it isn't for is we build this and we will build into this team who is adopting truly OpenStack to our hearts.

We are really taking most of the OpenStack processes such as CI pipelines and documentation, and process for reviews and blueprinting, and actually there is communication and manning and IRC, basically, they are throwing away all this traditional enterprise processes and it is, look. We are OpenStack. Right? We are embedded in OpenStack. It's fairly successful with such a large community around the world. Why not ... try to invent the wheel? Why not just follow and take the best there is? What really happened is Symantec is going through transformation.

A lot of people we're building the platform so they adopted the platform. The teams are moving to the devops world and the question becomes, "What is the best way to do it?" There is no silver bullet. I mean, we all know that. Devops is a loaded term, and everybody is doing it their own way, but really the message and what I'm seeing from our business unit looking at our teams. We are working in OpenStack and we warn them through the processes that we follow. They are really OpenStack flow. It was already scheduled and how do we do the documentation? How do we do designs?

It's the same model. You put a blueprint in. You get [inaudible 00:22:57]. It gets approved. You buy documentation you put it in gear and it gets approved. We have the same plus one, plus two, minus one, minus two processes. It's very straightforward once you understand and it's very powerful and I see a lot of eye openers on the product side when we are saying that makes sense. Why don't we move our process and to follow the same methodology OpenStack does? If you have a question look at OpenStack, that has been our default answer to a lot of businesses in Symantec.

Niki:                Obviously, there is naysayers in that process. How do you deal with the naysayers? That's always the trouble when you get smart engineers. It's like what do you do with the naysayers?

Yuriy:              It's always going to be like that. I mean, just like I said, there is no silver bullet, but you go with what you got and, like I said, now, we're not pushing it to anybody. It worked for us and the same things for that is something that we've learned, but I see it as a huge, what do you say, it helps jumpstart a lot of processes that stems more from traditional waterfall processes into more agile, devops cloudy world.

Jeff:                 What are some of the gaps you're seeing right now? You're talking about some where OpenStack kind of fits. Where are the open opportunities or where do you have to look outside of OpenStack right now or do you have any kind of a wish list?

Yuriy:              I do have a wish list, but that is, again, it's an enterprise versus smaller scale because with an enterprise you have all the enterprise use cases. It may not necessarily be OpenStack specific, but you do have to manage your hardware lifecycle management, but your hardware, having the onboard. You do have to ... Yes. We're looking at Keystone so we have our Hadoop infrastructure and something else, and how do you manage your identities, compliance security, all those things? I mean, they're not necessarily OpenStack-specific issues, but once you put this platform together it all has to come together because it needs to be ...

For operational efficiency, we need one ecosystem. It's not just getting gear. Somebody builds the gear and then here is OpenStack. It's not magic. It's a platform. That's where we're doing a lot of stitching yesterday to Randy's point is OpenStack from its source it's one thing, but to build a real platform, build a quote-unquote cloud and enterprise, everything else needs to come together, your charge backs, your metering and monitoring and your alerting all those capabilities needs to be in, in order to run a successful platform.

Niki:                It seems like you guys are kind of ahead of the curve probably more so than most companies I talk to regularly. What is the future of PaaS and containers look like for you guys?

Yuriy:              Good topic; good question. We are looking. We are looking at it right now to be honest with you. We're looking at different options. When we get [inaudible 00:26:38] is that going to fit? We are looking at do you have your fact sheet NASA's projects or you have cloud [inaudible 00:26:48] in the world. What I've seen which is very interesting is that you go back a year ago before the whole Docker revolution, whatever you want to call it, and PaaS introduction was quite tough because you have the right application for PaaS so I haven't seen it together too much.

I mean, some do better than others, but still. These containers what I see here right now is that our developers just to say, "Look. We are developing in containers. It works. Why do you have to change the way to do it? You're making it worse on your platform. Is it PaaS? Arguably not, but they expect a light sort of ... It's a light application lifecycle management. That's really what they want. They still want the same floor of here is my code. It went through CI. It went to production environment, and then somehow gets managed and then hopefully dies.

Now, manage that for me and that's what traditional PaaS would help with, but, now, having containers they still have to manage it for me, but, not necessarily lock me down to whatever process you want. It's a lot of push and we aren't doing it on our side. Here we have our bare metals. We have our VMs. We do have Alex [inaudible 00:28:19] and Docker as part of our platform. We have [inaudible 00:28:24]. We have Kubernetes, but, again, the challenge is that now you are going too wide and it become hard to manage. There needs to be something that still stitches it together. Is it OpenStack? Maybe. Or something needs to be on top of it. [inaudible 00:28:43] really need to [inaudible 00:28:45].

Niki:                Totally, yeah. It's like the whole Puppet versus Chef versus Ansible versus SaltStack.

Yuriy:              Yeah.

Niki:                Definitely, I mean the Rackspace there were groups that used all of them. How do you rank [inaudible 00:29:02] streamline processes when everyone is using different tools? How do you keep developers happy and keep them wanting to stay where they are and writing good code?

Yuriy:              It is because, I mean, if you think about ... I'm not sure how other organizations do it, but if I look at our platform and here is Symantec, yes. We are part of Symantec. Yes. We are solving for the business, but at the same time, we are internal private-public cloud, if you will. There is a fine line of the governance where we can control certain things, but other things we can't. When VM comes up ... I'll give you an example. In our initial deployment we had VMs that would come up and they will run configuration management to configure when they're under managed. A standard kind of exercise.

Guess what? Our customers come and say, "Why? This is my VM. What are you doing? I don't need you there." Build it up. Make sure it works and go away. Now, hey, we're not a public cloud. We can't really let you do whatever you want. We need the governance. We're a security company, but their point of view we need agility. Why are you forcing us to do something? The beautiful part about containers, now, we can have what we need in terms of controls. Yet, developers can have their part of the agility to move their port in and out without us being in the way.

Niki:                Are you finding that your internal users of this cloud platform that you're building are well familiar with writing applications for resiliency and redundancy? In other words, writing for cloud versus writing for say a VMware ESX environment?

Yuriy:              We're not there yet. It's part of the journey. Some things are better than others. We've still a ways to go to write a real cloud application. That's part of the challenge because people read what cloud is and they expect that cloud will do the magic. It will give them this ... It will solve. Just put it out there and it will scale. It will make sure it works. Where cloud is the other way around. It will break. Build your apps to make sure it breaks.

My favorite thing is one of our architects here when we go talk with internal customers and we talk about their vulnerability we always tell them, "Build for two nines. And they say, "What do you mean two nines? You can't have a platform that is two nines." Well, the platform is not, but when you build your applications, you have to assume it will go away, but there is a ways to go.

Jeff:                 That's right. Yeah.

Niki:                We have a large customer at Metacloud AKA Cisco I guess that intentionally walked up to one of their servers in one of their cabinets and just pulled the plug out just to see what would happen. It's like, man, it was good. It was good for us to know that our stuff was resilient enough to handle that, but at the same time, that's like you got to plan for that kind of stuff both at the OpenStack layer and at the application layer, and at the switching layer, and at every layer you can possibly think of if you're going to pro