Mastering Proactive AppSec

Intro:

What I recommend is basically start looking at open source stuff, free stuff, a lot of that stuff is available. And they have very good integrations with most of the modern frameworks, right? Start by leveraging that.

It might be a bit more painful to implement day to day. But once you have the dials and controls in place, as part of setting this up, you’ve built up the knowledge to assimilate a tool like this. Because when you go to a paid tool or whatever, it’s just a matter of sort of taking those feeds and taking those findings and running with it.

So I would say start off with open source. The other thing I would say is, pen testing is actually something which is, I don’t want to say underrated, but it’s very useful when you use it in the right way. You might have some critical applications in your organization.

And if you don’t have local pen testers on the team, I would definitely encourage you to sort of leverage some ways to either a third party vendor who can do pen testing for you. But even better is that if you could, participate in sort of a public bug bounty program or even a private bug bounty program where you’re exposing your application to a bunch of researchers that will give you tremendous bang for the buck that will actually almost compensate for your lack of having a tool out there, which is looking at everything because you have a bunch of hackers who are looking at your website and sort of hitting against it. I mean, just two of the suggestions to sort of start out.

Conversation:

Hello, and welcome to Secrets of AppSec Champions. My name is Chris Lindsey, and today we’re going to be speaking with Shashank. Today’s conversation is going to be around moving your application security program from a reactive to proactive state. Shashank runs the application security program at TripAdvisor.

Shashank, please introduce yourself.

Hey, Chris. Thanks. Thanks for inviting me. My name is Shashank. I’ve been running the application security program here at Tripadvisor for a good amount of, like, four, four and a half, five years.

And what’s really interesting is that I’ve had the opportunity to sort of build my own program, essentially, start from really nothing to build something which is, in my experience, proactive and sort of able to sort of weather the different issues that we face. So, really excited to talk about application security, very pumped up about application security. And so, yeah, looking forward to it.

Setting the Stage: Reactive vs Proactive Security

Awesome. Alright. So let me start with our synopsis so we kind of set the stage and let everybody know. Today’s conversation, just to let you know, Shashank is not going to speak at all about what’s going on at TripAdvisor.

We are going to actually be talking about just our history, what we’ve been through, and from that point of view. So let me get started. Alright. So your environment is on fire.

It does not matter what you’re doing. Anything that you do, any progress you’re trying to make is just not going anywhere. That’s a reactive environment. That’s a reactive program.

Moving to a proactive environment is where you’re actually seeing things get done. You’re actually moving the needle. You’re making things better for everybody. So in today’s conversation, we’re going to talk about what does a reactive program look like, but we’re also going to talk about the successes for a proactive program. With that, let’s get started. Shashank, let’s talk. Have you been in a reactive environment before?

Yes. I will say yes. And typically, what I see for most professionals is that when they’re really building a program from ground up, that’s where you start out with. Right? Because you don’t have all the capabilities deployed. You just don’t have it at that point of time. And so you’re really working yourself backwards from, right, having nothing at all to sort of then slowly building up the program.

In my mind, right? When I’m thinking reactive, it’s really kind of playing catch up with whatever’s going on in your environment. Right? And so whether it’s applications or scanning or defects and triaging these things, Right.

It’s all happening after the fact, right? You are maybe applications deployed to production and you’re just scanning stuff in production, right? You just don’t have any control over the software supply chain development cycle early on, like in the design phase or whatever. And so that’s how you start out.

And so I started off similarly and then sort of went my way backwards and sort of added more capabilities along the way in my whole journey, throughout this AppSec journey. And so, yeah, I would definitely say, if you compare and contrast Reactive to Proactive, one of the biggest changes you will find is that it’s much less load on the security team to sort of stay on top of everything that’s going on because a lot of the things you want to actually track would either be done through some of the tools that you deploy, but more importantly, through your relationships with the development organization at large.

And so I basically am of the opinion that you will automatically see the difference between a reactive program and a proactive program as you sort of make that transition, where you’re getting more bang for the buck for your program, and sort of lessening the stress around security because application security is vast and there’s a lot of things to look at. And so, yeah, you just need to make sure that your teams aren’t burnt out either.

Right. Application security is not an easy area. To be honest with you, a lot of leaders that I’ve spoken with over the years, it’s a struggle because there’s so much to it. And you’re right. When you’re starting your program, everything is reactive, because you know, a system will go down or something will happen, and you just won’t know why something’s going on. I don’t know why. And the networking guys come to you, and they’re explaining, hey, we’re seeing weird traffic, and you just have no visibility.

And as you start putting that tooling in place, you start having more and more visibility and all of a sudden you have that moment.

Have you been in that spot where you turned the tool on and you had that big moment?

So I will definitely so we’ve deployed a lot of tools, specific to AppSec in the past. And so, I think the one which I want to talk about is when we sort of, so when we look at developing a product, right? In a reactive environment, looking at what developers are doing day in and day out is not necessarily part of the equation, right? You’re very far removed from that process.

Implementing Tools for Proactive Security

But one of the initial things that we did was that, you know, we started, we installed a tool, which basically stands for like third party vulnerability is like a software composition analysis tool. And we shifted it left in the sense that it wasn’t in the CI/CD pipeline. And so that was, for me, the moment was really having the devs who were working in that environment see the vulnerability, know about the vulnerability, and take action on it at that point of time. And I was like, oh, I don’t have to do all this work later on in the game.

We can actually proactively fix all of this where, you know, you have a good relationship with the devs. So I think that scale that you get by not just by deploying any random tool, but by deploying to the right place to the right audience is something which will definitely help scale out your program in a very nice way.

Right. Well, and the benefit of doing an SCA as a tool first is it’s not only taking care of security debt that you’re unaware of, but it also looks at the developer technology debt. So from a development stand point, the developers will feel better knowing that we’re running or they’re running updated libraries and knowing that they’re current. So they’re getting all the features and all the benefits and any bugs that may have happened with any of the versions that they’re running.

In my background, I was a developer myself, and there were times where the library had bugs and you just worked around it. You didn’t really think, hey, let’s go grab the latest and greatest version because you didn’t know, can I just drop this in, and is it going to work, or is it going to break a lot of things? And so, you know, with a good tool, it gives you a lot more visibility into that.

Yep. That definitely makes sense. And, I’m just going to go back to, you know, if you’re thinking about return on investment rate, if I install this tool, what return am I going to get?

Installing tools in, like, the CI/CD pipeline is going to really be good dividends for you long term. Right? Because bear in mind, one aspect of it is sort of exposing to devs and sort of making sure that they’re fixing it. But another sort of hidden benefit of this, which you won’t see at the surface, is that devs are learning as they’re fixing these issues.

So they’re not repeating these mistakes again and again. And so what you eventually see with successful AppSec programs is that initially, you might see a bunch of stuff surface up. But then over a period of time, that whole thing matures and sort of you start seeing lesser bugs and you start seeing developers being more security aware and coding more securely, which in turn is exactly what you want from an AppSec coder. Right?

Building Relationships with Development Teams

Right. And, when developers know that you’re actually on their side and you’re trying to work along with them and help them be successful, then your program really it goes a lot smoother just simply because there’s not that friction of here’s something and you got it. Starting with an SCA is an excellent starting point. It’s low hanging fruit, but it also gives you the ability to go in there and work in their area to say, this is something that should be upgraded because you’re running thirteen versions behind or two versions behind or whatnot.

And so it’s not like you’re coming to them and saying you’ve got all these bugs and these problems and this bad security posture and you’re going to hurt the company. You’re saying, hey. Let’s do this right and keep a good, clean, tidy house on the development side.

And I’ll add one more point to this. Right? So, it’s when you’re coming in new, right, when you’re coming in from a reactive environment, you might not necessarily have that relationship with the dev organization at large. Right?

Like, maybe your app sec team doesn’t know who the dev organization is. Right? And so, these are the ways where, you know, you can actually once once you have the findings of maybe an SCA result, you go ahead and approach the teams and then strike up that conversation because that’s what’s going to lead to building up that relationship where, you know, they trust the security team coming. It’s not like we’re always coming to complain about the code or anything of that sort.

It shouldn’t be a mean conversation. And so, that’s also one thing I will highly stress is that even in your outreach to the development committee or whoever it is. Whoever the outreach is towards. Right?

Try to keep it specific and sort of make sure that, you know, you are, so we are the subject matter experts from a security perspective. So we should be aware of what the ramifications are, how to fix it, and, you know, what what the risk is and stuff like that. And then sort of have that conversation with the developers and and then win that trust. When you once you win that trust, trust me, right, they will be ambassadors of security, not just for their project or whatever, but they’ll developers talk to each other.

Right? And so they will be like, hey. You know what? This tool is pretty cool, and this is a new thing I learned.

And that can grow. Right? The culture. It’s more about security first culture, within the larger development.

Right. Well, you you hit a couple of key things that I want to, you know, kind of surface back up to. You talked about the developer and security relationships. If you have a good relationship, you’re going to be successful, just like you were mentioning.

The other thing you mentioned too is developer training. As they grow in their knowledge of security, they’re going to be learning. They’re going to know they’re going to create good best practices internally. They’re going to say, hey.

We’ve got a new version coming out of this application. We should go look at are there any updates to the to the libraries or dependencies?

The tools will do that for you, and some tools will even create the pull requests for you to kind of help you keep current. So that’s an excellent start. Where would you look for next in your program?

The Role of Continuous Monitoring in Security

I think the tool can get you so far.

Right? It it can sort of scan everything and create the pull request. Right? But the other side to that is that where where security’s work is not over is basically I I, always stress that, you know, we need to sort of have even though the devs are watching everything that’s happening in there, it’s visible.

It’s bubbling up to them. We need to have an overall visibility in terms of, you know, how many defects are they actually fixing? Are we reducing our risk profile and stuff like that. Right?

So, the follow-up aspect of that is constant. Right? It never ends where, you know, we’re constantly looking at what kind of criticals and highs and, whatever we want to track, how that’s trending. Right?

Because, it’s it’s not like we don’t trust the development team to fix it, but keep in mind, right, the developer’s primary job is to sort of ship features and sort of sort of write code to ship features, which are new products or new, or innovative innovations, in in that area. And so security might take a back seat in that kind of a situation in terms of priority. So it’s it’s equally important that, you know, yes, it’s very important to surface those to the developers, make them aware, train them and everything. But again, right, have that guiding guiding hand or whatever, advising them constantly following up with them and making sure that, yeah, you know about this.

What is the plan to fix it? Again, we don’t necessarily margin and say you fix it right now, but then we try to sort of understand, you know, what your risk appetite is and, you know, try to reason with them in terms of, you know, if you don’t fix this, this is the, this is the side effect of that. And, and more often than not, right? Developers are very reasonable, right?

They’re not very adamant in the sense that I wouldn’t fix something because I don’t want to fix it. It’s like if you have a decent enough conversation with them and if you explain it to them recently, I in my experience, I haven’t even seen a single developer who will say, yeah, I don’t want to fix it because I I don’t want to fix it. It’s like maybe I have other priorities or whatever. But, eventually, everyone fixes these things.

Right. Well, in in having a good foundation like that is is excellent for building a good successful program because, you know, the programs where the security team is really pushing the the development team to do stupid things. And and what I mean by stupid things is, you know, I have a list of forty vulnerabilities. They’re high.

They’re critical. You’re going to go fix this right now. The problem becomes in in in our pre call, we talked about this, so I’ll bring it up here. But it’s it’s a matter of, you know, your environment and and where does that vulnerability exist?

If it’s vulnerability that is sitting internally that is only accessible internally, that’s sitting on the command line or the terminal window inside Linux, it doesn’t matter if you have a high critical command execution or injection issue because you’re already there. And so having a good level, head on on your shoulders when you’re looking at your security program is key because going to the security or going to the developers with something like that, they’re just going to look at you and just shake their head, and and now you’re going to build up a wall. Whereas you come to them with something that’s actionable, something that’s actually relevant and and something that that’s critical.

If it’s really super critical, then you’re going to go to them immediately. You’re going to explain it. You’ll even show them why is this critical. Even show them possibly, you know, record it with a pen testing tool like Burp Suite or something and and and play it back to them.

This is why it’s critical. This is why we need to do something versus, hey. Guess what? I have a list of three hundred of them.

Can you get them done by, you know, next week?

Making Informed Decisions During Security Incidents

No. I completely agree with what you’re saying. I mean, it just, I’m reminded of the time when, like, for instance, a zero-day comes up comes along. Let’s take for instance, Log4j.

These are the kinds of times when these kinds of decisions need to be made. Because I remember, I think Log4j came over, kind of like the Christmas weekend last part of the year.

It was late November.

A lot of the folks would be out on vacation. And you don’t necessarily want to call everyone up and say, come online and sort of fix this issue for me because it’s a zero-day. If it’s public-facing, okay, maybe that’s the that bumps up the criticality. But, again, as you mentioned, if it’s sitting behind a VPN and if it’s guarded relatively well from that perspective, then maybe you can come back.

Maybe January or something, you can come back and take a look at it. Right? So it’s those kinds of decisions, right, where it’s not knee-jerk reactions to just you hear some news, something crazy is happening, and then you just rush to the dev desk and say, hey. We need to get this fixed.

Yeah. I completely agree with what you’re saying. It’s a trust relationship. It’s just not sort of, asking for something to get done for you.

Yes. So when you are in a proactive program and you’re actually working very well with the devs, one of the nice side effects that I saw when I ran my program was the development team would ask me questions about security. They would come to me and they would say, I’m working on a mobile app. I think we should probably have a lock screen. Is a thumbprint enough to unlock it? What do you think?

I think that’s a great sign that, you know, your program is actually maturing.

And I’ll preface it with this. Usually, and I’ll compare a reactive program versus proactive right here. Usually, in a reactive program, what you find is that the security team is actually reaching out to the developers. A lot of the outreach is security reaches out to engineering.

What happens in a proactive program, and it’s great when it happens, is that the dev team will automatically start reaching out to security for various matters. And that’s what happens when you mature a program. And so to that point, they will come to you with questions and recommendations on, like, something that you post.

And so I think it’s a great sign because, in certain areas specifically related to security, we have expertise in that particular area. And I think it’s very important that devs take in that feedback when they’re sort of designing various applications. Sure.

I mean, you can build everything on your own, but then us coming and doing a security review at the end, once it’s in production, once it’s published, that the cost to make those changes, I mean, everyone’s aware of the cost of fixing bugs along the production cycle. It just gets more expensive. Right? And so if you sort of reach out and again, this is this actually goes back, in for me, when I think of like, so SDLC, we have SDLC, but when I think of secure SDLC, these kinds of things are what really sort of help you along the way.

Right? Where if the devs are reaching out to you, with recommendations and advice and cons for a consultation or whatever, I think that’s perfect. I mean, that’s, that’s how products should be built.

And the truth is there. It’s not happening nearly enough, but when it does happen, it’s really nice. And I am all for it. So, yeah, I would give two thumbs up, if that’s happening in your organization.

The Role of Security Champions

Yes. And what do you think about security champion programs? Do you have any thoughts?

Very important. So again, this goes back to the relationship between security and development or engineering.

So for me, when I’m thinking of champions, I don’t sort of cast a wide net and say all developers are champions or something of that sort. I want my champion to be, surely security-minded, but really the way I look at champions is a way to scale out my operation.

I don’t have a hundred AppSec engineers on my team. And so it’s a bit most security teams are small. We have like five or six people within a team. And if you look at the application landscape, you’re looking at hundreds of applications and you’re looking like thousand plus, member dev organizations.

Right? And so champions are really the way you effectively scale out your operation because you’ve given them some sort of training. Usually, they have a security bent of mind, which is why they were attracted to the champions program in general. And so you’ve given them the right training.

You have frequent checkpoints with them, and it’s just not like, you know, you give them training and you forget about it. You need to have good assignments for them. Maybe we touch base very frequently.

We discuss various topics. What are you hearing on the ground? They would have exposure to some new things coming up and we’re like, yeah.

Tell us about it. Because we cannot be at all places at all times. And so basically, they’re just ambassadors of security within various points within the dev organization.

And so I think it’s tremendously important to have, or at least start thinking about having a good security champions program because that’s the only way we’re going to scale your program. That’s really the only way you’re going to get better visibility over what’s really happening out there. Because, yeah, you cannot really individually, you cannot have access to everything. So I think it’s hugely important.

Right. Well, in also with part of the, you know, having a champion program is also bringing developers into the security fold.

Exactly.

And, you know, it’s hard. There’s not very many developers who are also very deep in AppSec.

And so to be able to still have that developer come in, and as they grow and they learn in security, they’re also starting to be able to think about, you know, the criticality and how it doesn’t affect the environment. And, you know, they start thinking about and asking the questions. And the beautiful thing about having a champion program too is they’re sitting in the scrums, you know, for the development teams, possibly the senior tech reviews, and they’re asking the questions to QA. Hey. This is great. It’s passing.

I have a couple of checks I want to do, or maybe I want to do some automation like DAST as part of the QA process. That way, we kind of have an idea from a security standpoint there. Kind of an automation regression from a security standpoint, which is great. And then also, you know, when it all kind of humps together nicely, then then you know as the security group up above the four, the five, the six people running the whole program, it gives you the ability to sleep better at night. And as a matter of fact, I call it security’s boring at five o’clock on Friday because everything’s going right.

If security is not boring in your environment, you’re either not doing something right or you’re not visible to something very big that’s happening that you’re unaware of.

Hundred percent. I completely agree with you there.

Very cool. Can you think of anything else that would make a program that that’s proactive even better?

So program that’s proactive even better. Again, as you build your AppSec program, there are different levels of maturity as as you reach that.

And so, it’s great if you start out by sort of, even if you have like tools offline, which are scanning offline, and then you’re basically scanning production stuff, and then you sort of slowly shift left and you have CICD tools. But I think the real value, where you can actually proactively shift, push forward the agenda of security is if you have that mindset of product security, as a whole in general. And so that just doesn’t mean, you know, scanning where devs are sort of developing.

You need to move further left where you’re talking about design. You’re maybe working with product, say, for instance, and you’re working with, maybe things like threat modeling and sort of looking in architectures.

I mean, very early on in the life cycle of an application. And so I think that if you get that and if you get that right, where you’re in the right talking with the right stakeholders, and I’m assuming that, you know, you have a mechanism in place where, you know, maybe you have a seat at the table where you are getting that visibility into any of the new developments coming on.

So. I think having a conversation at that point and having a talk about security at the design phase, is ideal. And so I would definitely answer it that way, where, you know, the earlier you can actually talk security with the people who understand it, the better it is. I think that’s ideal for a proactive program.

Preparing for the Inevitable Security Incident

Awesome. And one thing I want to share, that worked very well with my program is bringing instead of going left going right, you’re actually going up, up to the executives and sharing and building reports and sharing the threat landscape with the executives, with your CSO, with your, you know, their boss and all the way up to potentially even the board and CEO. You know, that visibility, when they have the visibility into your program, they know, hey, we are really trying to be secure. They’re going to sleep better and they’re going to feel better knowing that people aren’t going to be able to steal the data, our data without real without us realizing it. And, you know, it’s one of those things where executives do not like surprises.

A security incident is a surprise for a lot of people.

And it’s one that you try to do everything you can avoid. And really the thing with security is it’s not a matter of if, it’s a matter of when. And when you’re doing the right security and kicking the can down the road, hopefully, it’s not going to happen under your watch. Hopefully, it’s not going to happen under the next person’s watch. Hopefully, it’ll never happen. But also, you know, when depending on what you are in the industry, what, you know, size-wise, you know, you can become a target or not.

You just always have to be prepared for the unexpected. And just all the different things that you’ve talked about and we’ve talked about today is really just kind of helping ensure that, you know, the basics, you know, starting with the low hanging fruit, the SCA’s, and then kind of working up and looking at the different things, the relationships. When you have that relationship with the developers, things just go a lot smoother. And it’s just it kind of makes for a great program.

Advice for Starting an AppSec Program

So let me ask you one final question, before we go.

And really, I’m just going to make this wide open. I know I kind of threw it out a little bit, but is there anything else that you want to throw in there for advice to somebody that’s small, somebody just getting started, or somebody that’s running a program. And think about, you know, kind of one of the biggest hurdles that you ran into and and maybe some a bit of advice that you might share to somebody and and say, hey. Welcome, Tapsec. Here’s something that might help you.

Right. And that’s equally important, right? I mean, it’s nice to sort of discuss about what mature programs do and you know, what kind of tooling is available. But when you’re coming in with apps, let’s assume you have nothing on the plate and you’re coming in brand new.

It’s, you’re starting the program from ground up. I think, one thing you should definitely look to, I mean, you should look to implement some sort of tooling to, let me see. You need to sort of implement some sort of tooling, which would be, which would assist you in sort of making sense of what kind of things you’re looking at.

And one of the first things you’ll see is right. You might not have the budget to buy the best tool out there and stuff like that. So what I recommend, is basically start looking at like open-source stuff, free stuff, that a lot of that stuff is available. And we have very good integrations with most of the modern frameworks.

Right? And so start by leveraging that. It might be a bit more painful to implement day to day. But once you have the dias and controls in place, you’ve actually, as, as part of setting this up, you’ve, you’ve built up the knowledge to assimilate a tool like this because when you go to a paid tool or whatever, it’s it’s just a matter of sort of taking those feeds and or taking those findings and making, and and and running with it.

Right? So I would say start off with open source. The other thing I would say is, you know, tools tools are great, but, you know, I would want to sort of get very good. I would want to emphasize pen testing is actually something which is, I don’t want to say underrated, but it’s very useful when you use it in the right way.

Right? So, you might have some critical applications in your organization.

And if you don’t have local pen testers on the team, I would definitely encourage you to sort of leverage some ways to, you know, either a third-party vendor who can do pen testing for you. But even better is that, you know, if you could, participate in sort of a public bug bounty program or even a private bug bounty program where, you know, you’re exposing your application to a bunch of researchers, that’ll give you tremendous bang for the buck. Right? I mean, that will actually almost compensate for your lack of having a tool out there, which is looking at everything because you have a bunch of hackers who are looking at your website and sort of hitting against it. I mean, just two of the suggestions to sort of start out. And then I’m sure that as you sort of start building the program, you will find your sweet spot in terms of what next to focus on and what to sort of, put in place in order to have good visibility and control over that.

The Value of Pen Testing in Security

No. I absolutely agree. I think, you know, to your point, pen testers are kind of fond of those necessity items at some point within your program because the value they bring, they can, you know, when you have a finding before you go to the development team and say, hey, there’s this critical finding, you can actually go pen test it, you can go attack it yourself, and just validate it. And if you can do it yourself and you have a good pen tester, then you know this is something that’s there.

And if you go to them and your development team and say, look, we have a finding. It’s critical. I need a hotfix. I need it today.

Here’s the rationale. Here’s the video or the details behind it.

They’re ninety-nine percent of the time going to say, thank you, we’ll get right on it, or, you know, that’s when you know you’re reactive when they’re saying go away.

I hope not. But, yeah, I agree with you. Yeah.

So, Shashank, thank you so much for coming. The conversation we had was absolutely amazing. The details, the insight, you know, knowing what a reactive program looks like, knowing what a good proactive program looks like is absolutely critical for success. And, you know, I appreciate you coming today, and I appreciate your insights.

Thank you. Thank you for having me.

Thank you so much for joining us today on Secrets of AppSec Champions. If you found today’s information valuable, hit that subscribe button on your Apple Podcast, your Spotify, or wherever you’re listening today’s episode.

Ratings and reviews are like gold for us. So if you’re feeling generous, please leave us a kind word as it helps others find our show. Until next time. Take care.