In 2025, web application attacks increased by 34% year-over-year according to Cloudflare's annual threat report. SQL injection, cross-site scripting, and path traversal remain the top three attack vectors — the same vulnerabilities that have dominated the OWASP Top 10 for over a decade. The reason is simple: applications are developed faster than they can be secured, and legacy codebases accumulate vulnerabilities faster than teams can patch them. A Web Application Firewall (WAF) provides a critical defense layer that catches attacks your application code misses.
Commercial WAF solutions like Cloudflare Pro ($20/month per domain), AWS WAF ($5/month + $1/million requests), and Imperva (enterprise pricing) are excellent but expensive at scale. ModSecurity 3, the open-source WAF engine maintained by Trustwave, combined with the OWASP Core Rule Set (CRS) 4.x, provides equivalent detection capability at zero licensing cost. This guide walks through a complete production deployment with Nginx.
Architecture: How ModSecurity 3 Works with Nginx
ModSecurity 3 is a complete rewrite of the original ModSecurity (v2). Unlike v2, which was tightly coupled to Apache, v3 uses a standalone library (libmodsecurity) with language-specific connectors. The Nginx connector is a dynamic module that passes HTTP transactions to libmodsecurity for inspection. The flow is:
Client Request
↓
Nginx receives the request
↓
ModSecurity Nginx Connector intercepts the transaction
↓
libmodsecurity evaluates rules in 5 phases:
Phase 1: Request Headers (before body)
Phase 2: Request Body (POST data, uploads)
Phase 3: Response Headers (before body)
Phase 4: Response Body (HTML, JSON)
Phase 5: Logging (after response is sent)
↓
If any rule matches with "deny" action → 403 Forbidden
If all rules pass → request continues to upstream
The five-phase processing model means ModSecurity can inspect both requests AND responses. This is critical for detecting data leaks — if your application accidentally exposes credit card numbers, Social Security numbers, or stack traces in API responses, ModSecurity can detect and block those responses before they reach the client.
Installation on Ubuntu 24.04 with Nginx
# Install build dependencies:
sudo apt update
sudo apt install -y build-essential git cmake libpcre2-dev libxml2-dev \
libyajl-dev libcurl4-openssl-dev libgeoip-dev liblmdb-dev \
libfuzzy-dev pkg-config nginx libnginx-mod-http-modsecurity
# If the packaged module is not available, compile from source:
cd /opt
sudo git clone --depth 1 -b v3/master https://github.com/owasp-modsecurity/ModSecurity.git
cd ModSecurity
sudo git submodule init && sudo git submodule update
sudo ./build.sh && sudo ./configure && sudo make -j$(nproc) && sudo make install
# Compile the Nginx connector:
cd /opt
sudo git clone --depth 1 https://github.com/owasp-modsecurity/ModSecurity-nginx.git
# Get your Nginx version and compile the connector as a dynamic module:
NGINX_VERSION=$(nginx -v 2>&1 | grep -oP '\d+\.\d+\.\d+')
wget http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz
tar xzf nginx-$NGINX_VERSION.tar.gz
cd nginx-$NGINX_VERSION
./configure --with-compat --add-dynamic-module=/opt/ModSecurity-nginx
make modules
sudo cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules/
OWASP Core Rule Set 4.x Installation
The CRS is a set of generic attack detection rules maintained by the OWASP Foundation. CRS 4.x, released in late 2024, includes major improvements: reduced false positives by 40%, better API and JSON support, a plugin architecture for extending functionality, and graduated paranoia levels for fine-tuning sensitivity:
# Download CRS 4.x:
cd /etc/nginx
sudo git clone https://github.com/coreruleset/coreruleset.git /etc/nginx/owasp-crs
cd /etc/nginx/owasp-crs
sudo cp crs-setup.conf.example crs-setup.conf
# Key settings in crs-setup.conf:
# Paranoia Level (1-4): Start with 1, increase gradually
# 1 = Low false positives, catches obvious attacks
# 2 = Moderate, catches more attacks, some false positives
# 3 = High, catches subtle attacks, requires tuning
# 4 = Maximum, catches everything, significant tuning needed
# Edit crs-setup.conf:
sudo tee /etc/nginx/owasp-crs/crs-setup-override.conf <<'CRS'
# Set paranoia level to 2 for balanced detection:
SecAction "id:900000, phase:1, pass, t:none, nolog, setvar:tx.blocking_paranoia_level=2"
# Enable anomaly scoring mode (recommended over traditional mode):
SecAction "id:900110, phase:1, pass, t:none, nolog, setvar:tx.inbound_anomaly_score_threshold=5"
SecAction "id:900111, phase:1, pass, t:none, nolog, setvar:tx.outbound_anomaly_score_threshold=4"
# Allowed HTTP methods (restrict to what your app needs):
SecAction "id:900200, phase:1, pass, t:none, nolog, setvar:'tx.allowed_methods=GET HEAD POST PUT PATCH DELETE OPTIONS'"
# Allowed Content-Types:
SecAction "id:900220, phase:1, pass, t:none, nolog, setvar:'tx.allowed_request_content_type=|application/x-www-form-urlencoded| |multipart/form-data| |text/xml| |application/xml| |application/json| |application/graphql|'"
CRS
Nginx Configuration with ModSecurity
# /etc/nginx/nginx.conf — Load the module at the top:
load_module modules/ngx_http_modsecurity_module.so;
# /etc/nginx/modsecurity/modsecurity.conf:
SecRuleEngine On
SecRequestBodyAccess On
SecRequestBodyLimit 13107200
SecRequestBodyNoFilesLimit 131072
SecResponseBodyAccess On
SecResponseBodyLimit 524288
SecResponseBodyMimeType text/plain text/html text/xml application/json
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogType Serial
SecAuditLog /var/log/nginx/modsec_audit.log
SecAuditLogFormat JSON
SecTmpDir /tmp/modsecurity/tmp
SecDataDir /tmp/modsecurity/data
SecArgumentSeparator &
SecCookieFormat 0
SecUnicodeMapFile unicode.mapping 20127
# Include OWASP CRS:
Include /etc/nginx/owasp-crs/crs-setup.conf
Include /etc/nginx/owasp-crs/crs-setup-override.conf
Include /etc/nginx/owasp-crs/rules/*.conf
# /etc/nginx/sites-available/your-app.conf:
server {
listen 443 ssl http2;
server_name yourdomain.com;
# Enable ModSecurity:
modsecurity on;
modsecurity_rules_file /etc/nginx/modsecurity/modsecurity.conf;
location / {
proxy_pass http://127.0.0.1:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
Eliminating False Positives: The Tuning Process
False positives are the number one reason organizations abandon WAFs. A properly tuned ModSecurity deployment should have zero false positives in normal operation. The tuning process follows a systematic approach:
# Step 1: Start in detection-only mode:
# In modsecurity.conf: SecRuleEngine DetectionOnly
# This logs all rule matches but does NOT block any requests.
# Run in this mode for 1-2 weeks to collect data.
# Step 2: Analyze the audit log for false positives:
cat /var/log/nginx/modsec_audit.log | jq '.transaction.messages[].rule_id' | sort | uniq -c | sort -rn | head -20
# Step 3: Create rule exclusions for confirmed false positives.
# Save exclusions in a separate file that loads AFTER the CRS rules:
# /etc/nginx/owasp-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
# Example: Your API endpoint /api/v1/content accepts HTML in the body,
# which triggers XSS detection rules (941xxx series):
SecRule REQUEST_URI "@beginsWith /api/v1/content" \
"id:1001, phase:2, pass, nolog, \
ctl:ruleRemoveTargetById=941100-941999;ARGS:body"
# Example: A specific parameter name triggers SQL injection detection:
SecRule REQUEST_URI "@beginsWith /api/v1/search" \
"id:1002, phase:2, pass, nolog, \
ctl:ruleRemoveTargetById=942100-942999;ARGS:q"
# Step 4: Switch to blocking mode:
# In modsecurity.conf: SecRuleEngine On
# Step 5: Monitor continuously. Check audit logs daily for the first month.
The key insight is that false positives follow a power law distribution — 80% of false positives come from 3-5 rules triggered by 2-3 application endpoints. Once you exclude those specific combinations, the remaining rules provide clean detection with zero noise.
Custom Rules for API Protection
The OWASP CRS is designed primarily for traditional web applications. Modern APIs need additional custom rules for threats specific to JSON APIs, GraphQL, and REST endpoints:
# /etc/nginx/modsecurity/custom-api-rules.conf
# Block requests without Content-Type on POST/PUT/PATCH:
SecRule REQUEST_METHOD "@rx ^(POST|PUT|PATCH)$" \
"id:2001, phase:1, deny, status:415, log, msg:'Missing Content-Type header', \
chain"
SecRule &REQUEST_HEADERS:Content-Type "@eq 0" ""
# Rate limiting: Block IPs making more than 100 requests per minute:
SecAction "id:2010, phase:1, pass, nolog, initcol:ip=%{REMOTE_ADDR}"
SecRule IP:REQUEST_COUNT "@gt 100" \
"id:2011, phase:1, deny, status:429, log, msg:'Rate limit exceeded', \
setvar:ip.blocked=1, expirevar:ip.blocked=60"
SecRule IP:BLOCKED "@eq 1" \
"id:2012, phase:1, deny, status:429, nolog"
SecAction "id:2013, phase:5, pass, nolog, setvar:ip.request_count=+1, \
expirevar:ip.request_count=60"
# Block oversized JSON payloads (prevent DoS via large JSON parsing):
SecRule REQUEST_HEADERS:Content-Type "@contains application/json" \
"id:2020, phase:1, deny, status:413, log, msg:'JSON payload too large', \
chain"
SecRule REQUEST_HEADERS:Content-Length "@gt 1048576" ""
# Block common vulnerability scanners by User-Agent:
SecRule REQUEST_HEADERS:User-Agent "@rx (nikto|sqlmap|nmap|dirbuster|gobuster|wfuzz|nuclei|burpsuite)" \
"id:2030, phase:1, deny, status:403, log, msg:'Vulnerability scanner detected'"
Performance Optimization
ModSecurity adds latency to every request. On a well-tuned deployment, this overhead should be under 5ms per request. Performance optimization strategies include disabling response body inspection for API endpoints that return large JSON payloads, using the @ipMatch operator for IP allowlists instead of regex, and excluding static asset paths from WAF inspection entirely:
# Skip ModSecurity for static assets:
location ~* \.(css|js|png|jpg|jpeg|gif|ico|svg|woff2|ttf|eot)$ {
modsecurity off;
expires 30d;
add_header Cache-Control "public, immutable";
proxy_pass http://127.0.0.1:3000;
}
# Skip response body inspection for API endpoints:
SecRule REQUEST_URI "@beginsWith /api/" \
"id:3001, phase:3, pass, nolog, ctl:ruleRemoveById=951000-959999"
A properly deployed ModSecurity + CRS 4 setup provides enterprise-grade web application protection at zero licensing cost. The initial tuning effort (typically 2-4 weeks of monitoring and exclusion writing) pays for itself the first time it blocks a real attack. ZeonEdge deploys and tunes ModSecurity for production applications as part of our security hardening service. Learn about our WAF deployment services.
Sarah Chen
Senior Cybersecurity Engineer with 12+ years of experience in penetration testing and security architecture.