The vulnerabilities you hear about aren’t really the problem much of the time.

I don’t want to dismiss the OWASP Top 10, because that’s what started the focus on Application Security.  And that’s important - it really is.  That said, we are kinda past that now, and while there are applications that harbor the flaws described in the Top 10, and many of the vulnerabilities described in the Top 10 that matter, the way that list is derived is not relevant to the way the world works anymore. If you dig into the data that makes up the list, you’ll find that 98% of it is unexploited static analysis findings.  Anyone who has spent time with static analysis will assure you that the only way static analysis if workable in a development environment is with triage.  And I’m sure you have guessed by now how much application owner triage goes into these results: 0%.  These findings might be legit, but they are probably not exploitable.

What’s number one on the top ten?  Injection.  I agree, injection sucks.  Anytime you can make an application run some code that the developer didn’t intend to run, you gonna have a bad day.

In reality? It’s a unicorn these days.  SQL injection, command injection, browser injection (which is explicitly a separate entry in the Top 10), doesn’t matter, vulnerability analysts just don’t find exploitable versions that much.  99.4% of the injections vulnerabilities cited in the data that makes up the Top 10 are from static analysis.  Did they break in? No, it was just possible.  Was there a mitigating factor - the fabled Security Onion? We’ll never know.

“OK, Bill, what ARE the Top 10 then?”

I don’t know. I’m an N=1 case, so I can just speak for me, and what I read.  That said, seems like a lot of people are using social engineering to get access to things.

“But Bill, that’s not an application security problem!”

Actually, it is. About a third of common application security vulnerabilities can be exploited with a social networking attack, and yet that’s the third we most frequently dismiss because it’s hard to demo during a BlackHat demonstration.  And guess what?  Those are the vulnerabilities I see most frequently.

But it isn’t vulnerabilities that are the problem.  It is the bugs.  Let’s look at the bugs.  We are always talking about the outcomes, not the causes, but here I want to talk about causes.  We can talk about the outcomes later.  Let’s look at a breakdown:

1. Out of date components: By far the biggest one I see is jQuery, which is arguably a big deal.  Mostly that is a DOM XSS problem, or info disclosure, and often the application isn’t using the feature that is exposed.  That said, there are a lot of holes here.  It is VERY hard to test all of the DOM XSS possibilities.  It’s far easier to just get rid of the sources and sinks.  
2. Information Disclosure: Not really on the Top 10 at all, but used by attackers to build the phishing messages that make the sysadmins answer emails. A6 Sensitive Data Exposure doesn’t cover it - this is an account number in the URL, or returning a password on a change screen.  I am talking about commenting out a block of JavaScript because it is “causing a problem on the backend” or leaving a developer name in the HTML comments.  
3. Cross Site Request Forgery: CSRF is a very complicated vulnerability with a very complicated exploit that has the simplest fix ever, and I have no idea why the fix isn’t part of every framework on the planet.  We have a session cookie already.  This is the problem. If we also have a session variable in the form post, the session can’t be forged (unless there is XSS).  Bang.  Done.  Why don’t we all do this?  Well, multihoming makes it hard, that’s for sure for starters, but I’ll have more on this later.
4. Cross Site Scripting: Yeah, OK, it’s in the Top 10 and it is still a problem  But you know where I see it? In the DOM! These never make it back to the server at all, no logging, no encoding, no nothing, what a pain in the butt.  And they still trash your CSRF protection.  XSS does not make me happy, which is why I put it on the report even if I can’t write a POC (which I rarely have time for).  Yes, I know that is grouchy and makes you dig through your JavaScript.  Sorry.
5. Insufficient cookie protection: There is absolutely no reason to fail to add SECURE and HttpOnly to your session cookie.  It’s like one line of config code. Oh, I’m sorry, you have a fancy JavaScript session management scheme?  Too bad, rewrite it.  It’s likely broken anyway (from the security perspective).  Let your servers manage session, stop doing the JavaScript thing when it comes to your sessions.
6. Vertical privilege escalation: The problem with VPE is that it requires some existing knowledge of the application.  In a 100% custom written application, that isn’t likely, aside from an insider attack.  The thing is, there aren’t that many 100% custom written applications.  Most projects start SOMEWHERE that is known. The authorization system is understood (along with weaknesses) or the framework has known page URLs (like WordPress) or something of the sort.  If the attacker knows where they are going and the authorization isn’t perfect, people can get to the administrator pages.
7. Unpatched servers: So I probably don’t need to say Equifax but … Equifax.  Seriously, if the Struts flaw doesn’t convince you that actively exploited flaws in your framework aren’t a risk, then nothing will.  When your vendor - open source or otherwise - tells you that you need to patch right now, you need to patch right now.  Not after your test cycle.  Not when management gives the OK.  Right now.  I’m a dev, I know it doesn’t work like that, but it has to, and soon.  This is getting bad.
8. Horizontal privilege escalation: When a developer keeps important account information somewhere that an attacker can edit it, and that information is used to decide what an attacker is looking at, they might be able to look at things they shouldn’t see.  Appropriate authorization solves this problem but it is very hard to do right,  It’s a lot better to not give the user a chance to change this value at all.  Insecure Direct Object Reference is the flaw in question, and it’s still out there.
9. Lack of Input Validation: A positive security model is a requirement for every application that faces the internet (and really internal ones as well, but that’s another topic). Every time you can, you should be checking the input against all of the possible inputs.  Can it not be negative?  Is it?  Reject it.  Is there a list?  Is the input on it? No?  Reject it.  Is it a free text field?  Fine.  Use the HTMLEncoder by OWASP.  Everyone (myself included) needed to do a better job looking for flaws in the validation of inputs, and removing them.
10. Weird stuff: There is so much weird stuff.  I got an application to give me account details because of a malformed USER AGENT.  Found the user’s role tacked onto the session ID in Base 64.  Discovered an application that parsed a word document - including a call the the template at a random IP on the internet - in order to allow editing.  From there to here, from here to there, funny things are everywhere.  If you find yourself thinking “Hey, that’s a weird neat compelling way to solve that problem” look for something simpler.  Complexity is the enemy of security.

I should probably talk about mobile.  The biggest thing that developers need to understand about mobile is that the compiled app is not invulnerable to being analyzed.  Don’t put anything in there that you wouldn’t want the user to have.  API keys, private encryption keys, and paths to functions the user shouldn’t have are three of the most common.  Just the other night someone stole the keys to send alerts from an app and sent random messages to all 100,000 users.  At 3AM.  I was not amused.

Anyway, this is just my take.  Again, I am not ripping on the Top 10, I love the Top 10.  It’s just useful to get the perspective from other-than-static-analysis companies.  Manual dynamic analysis has its place.  So in that spirit, I’d like to take you on a tour of how I go about performing vulnerability analysis - looking for the bugs that I have listed here. I’m no expert BUT I do it every day, so you might be able to glean something interesting out of my stories.

We live in a world where applications run the technology that we all use.  There was, once, a time where hardware was custom developed to solve certain problems, but these days we have general use hardware and applications designed to solve our problems.  Everything from apps on our phones to websites to alarm systems to the management screens for our internet gateways are applications, coded in common languages, using common protocols.

For every 100 lines of code, there are five security vulnerabilities.

The average application is 15,000 lines of code.

Let’s take a minute to talk about pentesting.  Pentesting, or “penetration testing” to expand the vernacular, is the art and science of finding a path through the security of a system in order to achieve a goal.  It is different from red teaming, because rather than making for a continuous series of tests like contemporary attackers would, penetration testing is a scheduled event that is designed to move from point A to point B.

Vulnerability analysis is different.  The goal is to take an application and find every single thing that could be used to circumvent the security of that application, then report on those items and provide a solution.  It is similar to penetration testing because it is a scheduled event. It is is easier than penetration testing because you are far less likely to go to jail for the night.  It is harder than pentesting because you don’t have to find one flaw, you have to find all of the flaws.

Quality assurance and vulnerability assessment have a lot in common. Both practices have the goal of making sure the application in question is as good as it can be.  Quality assurance is focused on the end user experience.  To that end, the both use a test plan. The plan has a starting state, an end state, and steps to get from one to the other.  With QA it is best if those tests succeed.  With vulnerability analysis, it is best if those tests fail.

That test plan is key.  In quality assurance, the business owners give a detailed description of how the application should respond under every circumstance.  If the user submits a valid application, it should be sent to the processing center.  If it has an invalid date, then this error will be presented.  If it fails in processing, then this message will be sent.

In vulnerability analysis, the test plan is determined by the attackers.  Whatever the flavor of the week is, it’s added to twenty years of attacks on the HTTP Protocol, language specifics, side channel attacks, and other weirdness.  The analyst will, rather than checking how the application responds to valid business requests, check how the application responds to this huge collection of known threats.  Not just the ones that work.  All of them.  The goal is the find all of the vulnerabilities.

It is true, not all of the vulnerabilities can be found. There are a lot of tests, and a lot of fields, and a lot of POSTs, and a lot of URLs.  They can’t all be checked  and fixed with machines (at least not yet). There isn’t the time or money to check them all by hand. Things will be missed, and that’s how we end up with 81% of breaches in the last 10 years having an aspect of application security involved. This is why we still have SQL injection, even in the age of ORMs.  This is why we still see CSRF even though it is a well understood vulnerability.  There are far too many legacy applications and far too few talented developers.

So why am I writing about this?  I performed my first vulnerability analysis in 2002.  That seems like a recent date to me - I wrote my first paid application in 1986.  But in the arena of application security this is a recent date.  The folks that wrote the Internet didn’t design it for security. That wasn’t the goal - sharing was the goal. I am writing this because I participated in the process of the web becoming the hub for commerce and communication - where security mattered.

And I, along with a boatload of others, failed miserably.

I got paged at 11PM on a Saturday in 1997 by a trigger I’d set up on a web server because the hard drive was full.  Long story short: the drive was full of German porn because of a SQL injection flaw I’d written into an application running on that server. Someone had broken in and set up a convenient FTP server.

At the time I didn’t know SQL Injection was even possible.

I’ll leave the path from then to now to the reader, but suffice it to say, I was the security guy on every project from then on.  I still strive to teach developers how to write more secure code - I’m the Security Track advisor to three conferences, and I speak to developers monthly about security awareness.  I train a thousand folks a year on secure coding standards.

That’s not why I am writing today, though.  There are way too few people checking applications for vulnerabilities, and OWASP isn’t making things obvious enough.  They have a greater reach than I, and that’s awesome, but I wanted to put together this book to lay out how I see vulnerability analysis in plain language, in hopes that it would help a few other folks get into the field.

What’s in this guidance certainly isn’t the only way.  It probably isn’t the best way.  It might be a bad way. I’m not sure, but it has worked for me, and I’m including things that you won’t hear in some breakdowns, like client management and report writing. Feel free to ignore everything, or take just the pieces that you like.  And send me feedback!  I’m more public than I should be on Twitter (@sempf) and Linkedin.  My Skype is pointweb.net if you want to tell me how awful it was privately. I’ll take your perspective in any form.