Cатсн²² (in)sесuяitу / ChrisJohnRiley

It’s been a while since I’ve really had a rant about the state of the penetration testing industry, and it’s certainly not because there’s nothing to rant about that’s for sure! There’s a whole heap of issues, so many in fact that at times they all seem to blur together into an impossible mess. I don’t know if there’s 99 problems, but I know for sure there are enough to keep us, as an industry, busy for years to come.

Here’s my, probably uneducated and naive, attempt to cover the big points. Who knows, this might even spur some discussion and maybe, just maybe, some solutions too.

The big issues (as I see it) .:

• Communication
• Relevance
• Accountability
• Standards
• Repeatability

Communication

I know it’s been said before, but it deserves to be said again. As an industry, we plain suck at communicating with the business. Chris Nickerson and others have spoken a lot about how we as security professionals don’t make things “business relevant”. The CEO doesn’t care if you get shell, he cares if you get the recipe for the secret sauce that drives his business.

As an industry we love technology and that shows in everything that we do. We spend 4 times as much time doing the technical side of things than we do writing about our findings, even though they’re the most important part! How does that compute? We also spend infinitely more time doing technical training than we ever do learning how to communicate effectively with the business. Just take a look at the range of classes offered from people like SANS, EC-Council, and others. There are elements of reporting and communication in some of the SANS classes, but nothing that spends nearly enough time really talking about the issues and helping people fine tune their skills. When’s the last time you saw people asking for peer review of their reporting templates or risk analysis?

Ideas: Better training for penetration testers on business communication. I’d love to see a 2/4 day SANS class on Technical reporting, communications and risk management!

Relevance

Some of the issues here go hand in hand with communication. Showing how that shell you got on the SQL server is relevant to the business, for example. However looking beyond this, there are other issues where testing lacks relevancy.

It’s hard to define what is and isn’t relevant for a customer, especially if they’re asking for a pentest or a redteam test, when you know they really need a vulnerability scan. Penetration testing really comes into its own for finding how your defenses stand up to a real attacker. If you have no defenses, no incident response plan, or god forbid, no security staff at all, then the client is just paying for a penetration tester to have fun for a while and come present at the end on how bad things are. There are plenty of testing companies that live off jobs like that! If we sell that kind of test to a customer, we’re doing them a dis-service!

Some issues are however outside the direct control of testers. Scoping for example, is and probably always will be an issue. Without educating the client on the real risks, the scope will always be an issue. Still, the only way to educate clients, is to build a relationship. Sometimes the first step is to play the game they want, build up that trust, before suggesting what might be a better fit for their maturity.

Ideas: Client guidance – from vulnerability scan through to highly focused penetration testing (Step, by step guide)

Accountability

Accountability in penetration testing is one of those tricky subjects that could either help to either legitimize or kill the industry dead. Let me explain my thinking here…

Current situation: The business pay for a penetration test, vulnerability scan, or whatever else they need. The testers come in, do there tests (whatever they might be) and write a (usually) poor report of the things they found. They leave, end of story.

X amount of time later, the business gets hacked through a vulnerability the testers didn’t find (or possibly even test for!). A penetration test is a moment in time, there are no guarantees made and no liability (at least none that I’ve seen tested in court). The worst that usually happens is the tester (company or individual) won’t get any further work from the business. For some companies, that’s the way they work anyway! As an industry we have our fair share of charlatans. Still, lets not go down that rabbit hole!

So how could accountability help? It should help to ensure that the testers are doing what the should be doing without needing to tie their hands and prevent creativity within the testing process. It could help businesses to know that they’re getting the testing quality they’re paying for.

Still, there’s another side to this too. Any accountability would have to be accompanied by increased costs, better standards to test against, and would have to be driven by the customers. There are a whole lot of issues here… far too many to write about right now. There’s a whole slew of things here that could cause problems, but without accountability of some sort, there’s little to drive businesses to continue throwing money into the black-hole of penetration testing.

Ideas: Name and shame – public feedback on testing companies

Standards

Yeah, I can’t believe I said that either. Standards stifle creativity and create boundaries that could reduce the effectiveness of some testing. Still, right now in penetration testing terms, we’re the equivalent of the wild west. We ride into town shooting our guns (mouths) off, saying we own this town, with nothing to back it up.

There are more than a few companies out there who think a Nessus scan is a penetration test. I’ve nothing against Nessus, as it serves it’s purpose very well. However, without some kind of standard to say what the minimum is, then some companies will always sell bad or unsuitable services to clients.

Standards would also be the first step in legitimizing penetration testing in the eyes of some businesses. InfoSec is still a young industry, and as fun as it is to hack things without needing to think about standards and compliance, it’s inevitable for the industry as a whole.

Ideas: Standard naming terms, Minimum testing standards for WebApp (OWASP Top10?) / Infrastructure

Repeatability

I guess it’s easy to understand, that without standards, accountability and the things I’ve listed above, there’s no way to make a test repeatable. If you send 5 testers to check an application, you’ll end up with 5 different reports, 5 different views on things, and 5 different opinions on the risk. How can businesses effectively track their security program if every time they test things, they get a different approach and a different test. Did we fix the vulnerabilities, or did the tester just not test it correctly?

Until it’s repeatable, it’s still too subjective to be 100% useful. I don’t want to reduce the creativity that goes into testing, after all that’s what makes this job so great. However without repeatability, we’re still just kids playing in the sandbox!

Ideas: Creation of test cases for discovered vulnerabilities

——

Well, there you have it, for better or worse. I don’t expect everybody to agree with these points. I also don’t expect these points to cover everything that’s bad or unmanaged in the penetration testing arena. What I do expect though is people to talk about things. Shout, stamp your feet, call me out and tell me I’m wrong… but do something other than nodding your heads and saying “yeah he’s right”.

Agreeing is nice. Arguing is good. Discussing is great… Doing is better!

With this in mind I’d like to setup a series of 5 podcasts to discuss each of the points above. As nobody wants to hear me talk about things alone for 30 minutes, I’m inviting you (yes, you…) to come on and talk. May your feelings known!

Shoot me an email at podcast [AT] c22 [DOT] cc if you want to be a part of it!