The Measured CSO – Alex Hutton
One of the most significant changes technology has wrought over the last decade is the current movement to use data and quantification as a means to better our everyday lives. In both our work life and leisure life, almost no aspect of modern life has escaped our desire to become better using evidence, data, and quantitative methods.
This talk discusses one method to help a Security Department build a better understanding of historically amorphous goals like “effectiveness, efficiency, secure, and risk” using data and models.
Where are we as an industry?
“… when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be.” – Lord Kelvin
This is the journey towards knowledge, and therefore security. We are at the point where we can’t talk about risk using high, medium, low. How would your investors feel if your CEO talked about profit as High, medium or low! We need to talk about things in a different way.
CVSS… “I use it every day, and I’m about to bash it!”
Where we’re at with our risk calculations:
- somewhat random fact gathering
- interesting, trivial, irrelevant observations
- little guidance to data gathering
First Mistake: Limiting ourselves
Security is an engineering issue… Looking at security only as a piece of the OSI layer.
Second Mistake: Blind leading the blind
Example: mobile malware is trending… this must be what we focus on. The FUD factory
Using the DBIR you can pull out more targeted and industry specific metrics that speak a lot more to the real threats. Looking at the DBIR it’s less than 1%. What we should focus on as an “industry” is not what’s hot right now!
mobile malware does not move the needle in out stats as we focus on organizazional security incidents as opposed to consumer device compromise
We’re dealing with complex systems… You can’t make point predictions in a complex system (Freidrich Hayek)
Correlation between CVSSv2 ratings and actual exploitations shows that even the highest rated CVSS vulnerabilities are not that widely exploited.
The measured CSO
The measured CSO must be more like W.E Deming…
The potential for improving the system is continuous and never ending… there is no perfect system. The only people who knows where the opportunities to improve are, are the workers themselves. There are countless ways for the system to go wrong.
Having workers are management speak the same language is important… having workers record and analyse statistical information helps to improve the system and evaluate changes easily. Everybody in the system has to be responsible for working towards improvement.
How many of us spend an hour doing statistical analysis on the other 38 hours of work we’ve done!
A measured CSO:
- Relies on metrics, data, intel for good decisions
- Invests in improvements to people, process and technology
- …
To provide the best and least-cost security for shareholders, and continuity of employment for his workers
- We as an industry, know that “best” and ” least-cost” are not necessarily contradictors
- We also have a HUGE continuity issue
Extending something like VERIS is incorporate controls data can assist a measured CSO in understanding where they stand. Using map reduce (HADOOP) this information can be modeled and look for IOC. The key to this is enriching the data with as much metadata as possible.
Framework <–> Models <–> Data
The Metrics and models that “defend” against threat patterns
Mobile malware might not be an issue now, but we need to plan, build, and manage to ensure when it is an issue, we have things already in place.
A Micromort… a one in a million chance of death… we can apply that
We’re bad at combining all those metrics… overweight, on drugs, and doing something stupid.
Becoming measured
What does that mean? What do we need?
Most metrics programs are gathering of some information without any context.
A metric is like a lego piece. It has no context until you build something with all the lego pieces you have.
How do you get context?
Goal, Question, Metric (GQM)
- Execution: Define goals
- Models: Question how this can be measured
- Data: Define metrics that answer the question
The measured CSO creates a scorecard of KRI’s and KPI’s that he can use to evaluate where they currently stand
Framework for GQM –> NIST CSF (Cyber Security Framework)
Links:
Cатсн²² (in)sесuяitу / ChrisJohnRiley 