Measuring web security mitigations

Summary: This past weekend I spent some time implementing a prototype for a web security mitigation, and I also spent some time thinking whether it was worth implementing as a web platform feature or not. In this blog post, I want to share how I approached the problem, which I hope you find useful, and to hopefully get your feedback about it too.

I've seen amazing progress on controlling the effects of XSS by adopting inherent safety on software engineering (a term which means focusing on completely eliminating the hazard) and I'm fairly confident that is today's most effective way to tackle it. However, there is always value in evaluating other types of controls beyond pure prevention - perhaps moving on to ways to minimize or contain its risk.

The way I see the problem is that the value of a mitigation can be measured by:

• Impact - How many vulnerabilities was this mitigation designed to control?
• Difficulty - What is the cost to adopt this mitigation on a system?

Measuring difficulty is somewhat easy, one should just try to apply the mitigation to real-life applications and see how difficult it is, however, measuring impact can be really difficult on a complex system.

The way this problem was approached in other fields was by measuring mitigations and controls across four metrics (wiki):

• Moderation - How much are we limiting the impact of the problem.
• Minimization - How much are we minimizing the number of problems.
• Substitution - How much are we replacing risks with safer alternatives.
• Simplification - How much are we removing problems, rather than adding complexity.

So, a naive way to look at this problem, is to evaluate the impact a mitigation has across these four metrics.

For example, I am a fan of Suborigins, an idea that aims to limit the impact a single XSS vulnerability can have by creating a more fine-grained concept of an origin. Suborigins is a good example of Moderation. It does not reduce the number of XSS vulnerabilities, it just makes it so that their impact is significantly reduced. On the other hand, we have the Angular sandbox - it aimed to limit the impact of the problem, but doing so effectively was extremely difficult, and eventually, the sandbox was removed completely.

Another good example is Minimization, and a great example of this is X-Content-Type-Options and X-Frame-Options. These are HTTP headers that allow a site owner to opt-out of behavior that can cause clickjacking, or some types of content sniffing vulnerabilities. If a website owner deploys these headers across their whole website then the amount of places that are likely to be affected can be drastically reduced. On the other hand, we have browser XSS filters and Web Application Firewalls. After many years, I think our industry reached consensus that they are not real security boundaries, and we largely stopped considering bypasses as security vulnerabilities.

Then we have mitigations that website owners can take that can Substitute a risky behavior with a less risky alternative. A good example for this is the use of JSON.parse instead of eval(). By providing a safe alternative to parse JSON content, the browsers have allowed website developers to write code that parses structured data without having to fully trust the data provider. On the other hand, we have DOM APIs (createElement / setAttribute / appendChild) as a replacement for innerHTML. While the use of DOM APIs is really safer, it's also so much more difficult and inconvenient that developers just don't use it - if the alternative is not easy to adopt, developers just won't adopt it.

And finally, we reach Simplification. A great example of a good simple API are httpOnly cookies, it does what it says, it restricts cookies so that they are available over HTTP only, and not JavaScript making credential stealing (and persistence) really hard in many cases. On the other hand, Ian Hixie eloquently explained the complexity problem with CSP back in 2009:

First, let me state up front some assumptions I'm making:

• Authors will rely on technologies that they perceive are solving their problems,

• Authors will invariably make mistakes, primarily mistakes of omission,

• The more complicated something is, the more mistakes people will make. 

I think a valuable lesson (in retrospect) is that we should aim for baby steps (like httpOnly) that provide obvious simple benefits, and then build up on that, rather than big complex systems with dubious security benefits.


And that's it =) - The purpose of this blog post is not to make a scorecard of different mitigations and their merits, but rather to propose a common language and framework for us to discuss whether a mitigation is valuable or not. Hopefully, so that we can better focus our efforts on those that make the most sense for the internet.

Thanks for reading, and please let me know what you think below or on twitter!