How to test the WAF?
A couple of weeks ago, the Vulners team posted a comparison of several popular WAFs. Catching myself thinking — “how to evaluate the quality of its work?”, I decided to analyze in more detail the topic of security tests and criteria for evaluating the Web Application Firewall. The article will be useful, first of all, for those who are interested in the topic of web security, as well as for the happy owners of WAF.
Criteria for evaluation
When comparing various solutions, we are used to focusing on performance indicators, performance stability, ease of setup, management, updating and scaling. In the context of WAF, this list should be supplemented with the accuracy of detecting attacks — the number of False Negative and False Positive.
Zoo from UTF-8
To understand how many subtleties there are when parsing requests, consider the feature of working with UTF-8 as an example. If in the search box of the site we type something in the English layout, then, most likely, the Basic Latin set will be used.
In this case, the request that will be transmitted to the web application looks like this: GET /?s=test
Now let’s try to use the character set from Halfwidth and Fullwidth Forms (looking for the same test, only in a different encoding).
It looks like we used a space between characters, but in reality we haven’t.
Request (in URLencode): GET /?s=%EF%BD%94%EF%BD%85%EF%BD%93%EF%BD%94
In this case, both queries (with different character sets) will lead to the output of the same result — an object containing the entry “test”, although we use a different character set.
At first we thought that the web application itself (tested on WordPress, but will work on others) normalizes the data to Basic Latin, but in the end we found out that the data is transferred without normalization directly to the database. There are limitations associated with the use of Halfwidth and Fullwidth Forms — it cannot be used when writing a SELECT statement, but it can be used, for example: LIKE '%Text in Halfwidth and Fullwidth Forms%'. This character set can be used not only when writing SQL queries. There are examples of its use when searching for XSS, etc.
This feature allows WAF traversal if it does not normalize characters to Basic Latin. There are many workarounds out there, and it would be great to create a tool to test them all at once.
WAF Bypass: toolkit
On github, you can find many solutions that allow you to test WAF, there are even special accounts that collect such tools, for example waflib.
In addition to the main unit-test, which we use to test the Nemesida WAF before each release (the test for each OS version takes about 30 hours), we have developed own waf-bypass tool, which allows us to evaluate the accuracy of detecting attacks. But it is always interesting and useful to test the work of the product using third-party solutions, so as the second (and within the framework of this article — the main) tool, I chose Go Test WAF, hosted in the wallarm user git repository.
To use waf-bypass written in Python3, you need to clone the project, install the dependencies and run the script:
# git clone https://github.com/nemesida-waf/waf_bypass.git /opt/waf-bypass/
# python3 -m pip install -r /opt/waf-bypass/requirements.txt
# python3 /opt/waf-bypass/main.py --host='example.com'If you follow the documentation, Go Test WAF can be run from a docker container, which is what I did:
# docker build . --force-rm -t gotestwaf
# docker run gotestwaf --url='http://example.com'If we compare tools based on their use, then Go Test WAF is faster (due to multithreading) and contains a small set for testing False Positive. We will take this into account in the next waf-bypass releases.
I see the goal, I go to it!
I will test 2 versions: Nemesida WAF (full with machine learning) and Nemesida WAF Free (free, but only with signature analysis).
In the Vulners review, in the final table, the maximum WAF score obtained by testing the Go Test WAF was 83.06%, while ModSecurity scored 49.82%. It’s interesting to see how much Nemesida WAF and Nemesida WAF Free can score.
Nemesida WAF Free bypass
Let me remind you that the analysis in Nemesida WAF Free is performed only by the signature method, while this version has a well-written and automatically updated own signature base (rlinfo.nemesida-security.com), as well as a Nemesida WAF Cabinet (also installed locally) — a web interface for visualizing events. Nemesida WAF Free is presented as a dynamic module that can be connected both to a new instance of Nginx and to an already installed one (or compiled with its own flags). Updates Nemesida WAF Free from the repository and is available for Debian, Ubuntu and CentOS, “quick start” will take 5–7 minutes for advanced users.
Run Go Test WAF, see the results:
45.47% and 0 false positives is pretty good, considering that some of the tests contain Base64 payloads that are not decoded in Nemesida WAF Free. Now, with a dexterous movement of the hand and without changing the settings of Nemesida WAF Free, let’s check it using waf-bypass:
325 skips and 1054 blocked payloads. We try to remember the results and move on.
Nemesida WAF bypass
In addition to the data normalization functionality, Nemesida WAF allows decoding requests in various encodings, including Base64, so it was especially interesting how fast it can type. Let’s start with waf-bypass, run it, see the results:
4 passes, other requests are blocked. It’s great, but still waf-bypass is our development, so it’s very interesting what results will be obtained when testing using the Go Test WAF. Run Go Test WAF, see the results:
The check took a little longer, but the scores are 2x better — 94.45%, and 11.39% higher than the best result in the Vulners review. There are almost no False Negative, but out of 8 False Positives, 5 are blocked, but I wanted 0, so we’ll figure it out. First, let’s see the contents of all 8 FP requests:
- union was a great select
- 'h2<h1'
- D'or 1st parfume
- "1) a-b=c"
- john+or@var.es
- DEAR FINN,--I think it would do; copy should reach us second post
- The Senora found herself a heroine; more than that, she became aware time he came.Of these, mistakenly blocked:
- f971e9ace6=union was a great select
- 24d85a0990=D'or 1st parfume
- 48105a7f77=john+or@var.es
- 18196a6191=DEAR FINN,--I think it would do; copy should reach us second post
- e7e5421304=The Senora found herself a heroine; more than that, she became awareTo understand why the requests were blocked, you need to understand how the machine learning module in Nemesida WAF works — the models are built using 2 sets of training samples — the attack base and the base of conditionally pure traffic. The fewer real queries are included in the training sample, the more False Positive will be. In this case, such or similar queries did not come across in the training set, in addition, they contain signs of an attack (for example, ––, or union … select, or ' or …), so they were blocked.
Using the Nemesida WAF Signtest retraining module, we export False Positive in one click, after which they will be included in the training set and will not be taken into account — both themselves and other queries similar to them.
Run the test again:
Despite the fact that the content of the requests has a dynamic content of 24d85a0990 in 24d85a0990 = D’or 1st parfume, there are no False Positives, and the number of skips has not changed. The scoring has become smaller, but should have increased — probably an error in the Go Test WAF itself.
In fact, there were only 3 FN, and judge for yourself about their criticality:
GET /AEAEAFAEAEAFetcAFpasswd
GET / QUIT /
GET /foo_bar=foo bar=foo[barfoo[barAttack visualization: graphs, search, reports
Regardless of whether you use Nemesida WAF or Nemesida WAF Free, you can connect your Cabinet to any of them — a web interface that allows you to visualize the work of the modules:
There are other sections in Nemesida WAF Cabinet— information on brute-force attacks and SMS flooding, statistics on the vulnerability scanner, etc. You can also download the general report in PDF or CSV format. The Cabinet is written in Django and is installed locally, just like other modules.
Conclusion
As I wrote above, there are several indicators for assessing the quality of the software, for WAF one of the main criteria is the number of False Positives and omissions, and it was very exciting and useful to perform checks using third-party bypass, for which special thanks to the developers of Go Test WAF.
In addition, such tests allow you to clearly understand the difference between signature analysis and machine learning, the need to use various normalization and decoding mechanisms. But if for some reason it is not possible to use a commercial product, you can always use ModSecurity, NAXSI or Nemesida WAF Free with a beautiful and convenient Cabinet.
Nemesida WAF Free is available as installation distributions for Debian / Ubuntu / CentOS, as a Virtual Appliance for KVM / VirtualBox / VMWare and an image for a Docker container. More information can be found here.
Thank you for reading, if you have any thoughts — you can discuss in the comments.
