Hey all, i’ve decided I should probably setup something else to help block nefarious IP addresses. I’ve been looking into CrowdSec and Fail2Ban but i’m not really sure the best one to use.
My setup is OpnSense -> Nginx Proxy Manager -> Servers. I think I need to setup CrowdSec/Fail2Ban on the Nginx Proxy Manager to filter the access logs, then ideally it would setup the blocks on OpnSense - but i’m not sure that can be done?
Any experience in a setup like this? I’ve found a few guides but some of them seem fairly outdated.
Edit: thanks everybody for the great info. General consensus seems to be with crowdsec so I’ll go down that path and see how it goes.
You must log in or register to comment.
I’ve been meaning to check out crowdsec because it seems to fit my niche usage. Wuzah seems VERY powerful and something I could likely use at work so that’s an advantage but very complex. Fail2ban is good at what it does but very simplistic and would require a lot of individual instances that would seem like a bear to maintain. CrowdSec seems like it’s in the Goldilocks zone somewhere in the middle. Pretty powerful, not terribly hard to manage, and not too difficult to install. But I haven’t done anything with any of them yet so I’m not very much help. I am curious what you go with though.
CrowSec, it’s not one that you mentioned but once they locate the source of the malicious traffic, they send out a murder of trained attack birds to resolve the issue.
Nah, that one conflicts with my IPoAC networks unfortunately :(
RFC 1149 compatibility is expected in an upcoming release, keep an eye out.
Fail2ban unless you need the features that crowdsec provides. They are different tools with different purposes and different features.
I thought crowdsec does everything fail2ban does in addition to global block lists?
Fail2ban is a Free/Open-Source program to parse logs and take action based on the content of these logs. The most common use case is to detect authentication failures in logs and issue a firewall level ban based on that. It uses regex filters to parse the logs and policies called jails to determine which action to take (wait for more failures, run command xyz…). It’s old, basic, customizable, does its job.
crowdsec is a commercial service [1] with a free offering, and some Free/Open-Source components. The architecture is quite different [2], it connects to Crowdec’s (the company) servers to crowd-source detections, their service establishes a “threat score” for each IP based on detections they receive, and in exchange they provide [3] some of these threat feeds/blocklists back to their users. A separate crowdsec-bouncer process takes action based on your configuration.
If you want to build your own private shared/global blocklist based on crowdsec detections, you’ll need to setup a crowdsec API server and configure all your crowdsec instances to use it. If you want to do this with fail2ban you’ll need to setup your own sync mechanism (there are multiple options, I use a cron job+script that pulls IPs from all fail2ban instances using
fail2ban-client status, builds an ipset, and pushes it to all my servers). If you need crowdsourced blocklists, there are multiple free options ([4] can be used directly byipset).Both can be used for roughly the same purpose, but are very different in how they work and the commercial model (or lack of) behind the scenes.
Neither. Use wazuh. You’re welcome.
I actually refrain from using Crowdsec since we found ourselves with a friend banning each other for no known reasons. (I swear I’m a good boy)
As you probably know the crowdsec bouncer doesn’t directly parse logs or do checks like F2B filters. It queries the crowdsec LAPI for decisions and applies them. The “allowed” or “whitelisted” IP logic is handled at the Security Engine or LAPI level, not by the bouncer itself.
You can whitelist an ip in
/etc/crowdsec/whitelists.yamlor even whitelist decisions in the whitelist.yaml as such:name: private-ips description: Whitelist local and private IPs whitelist: reason: "Allow local and private IPs" ip: - "127.0.0.1" - "192.168.1.0/24" cidr: - "10.0.0.0/8"Then issue
sudo systemctl reload crowdsec. Kind of the same concept as F2B’signoreipoption. If you are using Tailscale to administer the server, then it’s easier to whitelist. IIRC, you can usecscli decisions add --type whitelist --ip 192.168.1.100 --duration 1ybut it doesn’t add them to the whitelist.yaml. Instead it keeps them in crowdsec’s database managed by LAPI. To undo:cscli decisions delete --ip 192.168.1.100 --type whitelisthttps://docs.crowdsec.net/u/getting_started/post_installation/whitelists/
Care to elaborate? This seems kind of insanely specific.
Also, if you’re using fail2ban, the same thing would happen.
I don’t have much to elaborate on ^^’ but yeah, could have been an hyper specific case but that was my experience with it. I assumed my ip was banned on the crowd or something like that and even if my friend unbanned me twice, the ban came back. Don’t know what really happened for sure.
Ok…but crowdsec bans abusive IPs. Are you saying your actions got you banned for some reason?
Also, whitelist first. Ban second.
I truly don’t think I did anything to get banned. The only thing “non-standard” I do, is having a seedbox
Crowdsec is much more efficient than fail2ban. Fail2ban is a lot of old single-threaded Python code with inefficient log parsing/tailing routines. Crowdsec is a more modern Go codebase.
If you’re looking at old-school solutions, there’s also DenyHosts.
Crowdsec will block external, public, IPs
Fail2Ban will block login attempts (ie from anywhere)
I have a similar setup with pfSense, pfBlockerNG, HAProxy, etc, but I keep F2B running on my DMZ server in case something is ever compromised as it’ll block / slow down anyone trying to move around the network.
Crowdsec blocks login attempts too.
From the guy that has been accused of going overboard on security measures, I use both. It just depends on your setup tho. On a low resource server, I would pick crowdsec as it covers more ground than F2B. Running two log parsers does use more resources. ~ my 2 cents
I had fail2ban running for several years before switching to CrowdSec late last year. They both work in a similar fashion and watch your logfiles for break in attempts. With the small difference that CrowdSec also lets you use blocklists from the “crowd” to block malicious actors before they even get to try their luck on your machine(s).
I’m using CrowdSec with Traefik and nftables. But there are some bouncer plugins for nginx and OpnSense, too.
I just followed their example configurations for Docker, Docker Compose and then started tinkering with the config until everything worked as desired.
Thanks those links were helpful to get me on the right path. I like that there is a plugin for Opnsense directly and has that central LAPI, because I’d need something similar if I was to use f2b.
I have traefik running with all config done via the docker compose files and I just couldn’t figure out how to get the bouncer middleware to work without causing problems. Doesn’t help that most examples seem to be based on the static yaml based config so I’m trying to convert jt. Would appreciate anyone who might know of a resource that explains with docker compose environment tags.
I also have middle ware for things like authentik which complicates things.
In the Traefik static configuration (usually
traefik.yml), add this to load the CrowdSec plugin:experimental: plugins: crowdsec-bouncer-traefik-plugin: moduleName: "github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin" version: "v1.4.2"(The name for the plugin is defined here as
crowdsec-bouncer-traefik-plugin.)Then, in your dynamic configuration, add this (I’ve used a separate file
dynamic_conf/050-plugin-crowdsec-bouncer.yml):http: middlewares: crowdsec-bouncer: plugin: crowdsec-bouncer-traefik-plugin: CrowdsecLapiKey: "...YOUR CROWDSEC LAPI KEY HERE..." Enabled: true(The name for this new middleware defined here is
crowdsec-bouncer. It uses thecrowdsec-bouncer-traefik-plugindefined in the previous step. Make sure these names match.)You can get the LAPI key by registering a new bouncer in CrowdSec.
And, finally, make sure all incoming traffic routes through the bouncer plugin. You can do this individually, or in general via the static config:
entryPoints: websecure: address: :443 http: middlewares: - crowdsec-bouncer@file - secure-headers@fileThe middlewares are processed top to bottom.
Any change to the static configuration requires a restart of Traefik to become active.
In addition to the explanation you got from the other user: once you’ve set up the bouncer middleware in the configs (don’t know if there even exists a good way to do that outside of the configs files), you simply assign the middleware in the compose file as usual.
I’ve been using crowdsec … but I’ve yet to see anyone banned but myself so far. Is everyone else having to write tons of whitelist parsers? I could whitelist my IP but I feel like that’s sidestepping the issue and doesn’t address friends/family also getting banned, coffeeshops, etc.
Feels like I’m missing something as so far it’s been quite a pain to configure
Where did you have it setup? Is your proxy configured to forward the real IP?
It’s set up on the same box as my caddy install. I believe it’s getting passed the real IP because that’s what gets banned, and what I type in to unban it.
It just sees normal operations as http probing. Like if some other service goes down, my GetHomepage will then 404 and that’s seen as probing. It bans surprisingly quick. Even after just one or two events (normal for someone just visiting the homepage) it’ll just kick em right out
I’ve been having to inspect every alert and hand write whitelist parsers to whitelist 404s or whatever it may be for that app. Slowly accumulating a workable collection… but seems like I’m missing something as no one else seems to complain about this in threads like these
Another example is my brother got banned for normal audiobookshelf usage. He just thought the server was buggy. It was just blocking him without us really noticing or thinking much of it at the time. Not great
Crowdsec with a central LAPI server. You should install it on the servers themselves to monitor the application logs directly. Then every bouncer(firewall, router, edge device) connected to the LAPI will all block the same IPs. I got sick of repeat offenders and upped the ban time to 1 year in hours.
Awesome that makes a lot of sense, cheers. So I’ll install the Crowdsec agent on the Nginx Proxy Manager, and potentially also on the servers.
No problem. It’s a great piece of software. I have it monitoring logs for nextcloud, vaultwarden, mailcow(postfix & dovecot), basic nginx proxies (just to be safe and for rate limiting). I have 4 OPNsense and 1 Debian bouncers.
I had an issue with so a note about setting up the bouncer on OPNsense. If you have the LAPI on a different machine you can currently only connect OPNsense to it using the command line. The LAPI options in the web interface are for defining the interface to bind to and run the LAPI on OPNsense itself. Which isn’t an issue, I just wanted it on a VM so it’s easier to keep online instead of it going down if the OPNsense it’s on fails. Plus I like to keep SSH disabled on my OPNsense devices and spend a bit of time using cscli on the LAPI VM from time to time.
Cheers, I’ve since discovered that’s is “bouncers” that I want on the endpoints I.e on my Nginx Proxy Manager. I’ll just use the LAPI on the Opnsense box for now I think.
I’ve recently enabled banning whole subnets if more than 3 malicious actors from that subnet are on the blocklist. This is great for all those DigitalOcean droplets and other cheap hosters used by those people…
I’ve been thinking about going this route. What size subnet are you banning? /24?
Only thing stopping me is I selfhost email and don’t want to ban say a whole subnet from Microsoft/Azure and end up blocking the outgoing servers for O365. I’m sure I can dig around and look at the prefixes to see which are used for which of their services just haven’t had the time yet.
I let CrowdSec determine that. I’m seeing
/13,/12and even/10in my decisions list. All seem to be Amazon AWS ranges.Wow, those are big networks. Obviously I suppose in case of AWS it doesn’t matter as no human visitor (except maybe some VPN connection?) will visit from there.
As someone who bans /32 IPs only, is the main advantage resource consumption?
Im a lazy mofo. I use fail2ban since SWAG has one built in.
Crowdsec if you have many instances that need to report to each other.
If you just have a single instance and care to configure f2b for those services, then it’s fine. I would suggest incorporating the use of public blocklists though.
I’m currently going through a similar situation at the moment (OPNSense firewall, Traefik reverse proxy). For my solution, I’m going to be trial running the Crowdsec bouncer as a Traefik middleware, but that shouldn’t discourage you from using Fail2Ban.
Fail2Ban: you set policies (or use presets) to tempban IPs that match certain heuristic or basic checks.
Crowdsec Bouncer: does fail2ban checks if allowed. Sends anonymous bad behavior reports to their servers and will also ban/captcha check IPs that are found in the aggregate list of current bad actors. Claims to be able to perform more advanced behavior checks and blacklists locally.
If you can help it, I don’t necessarily recommend having OPNSense apply the firewall rules via API access from your server. It is technically a vulnerability vector unless you can only allow for creating a certain subset of deny rules. The solution you choose probably shouldn’t be allowed to create allow rules on WAN for instance. In most cases, let the reverse proxy perform the traffic filtering if possible.
I did have that same thought actually, with opening up opnsense to be modified. But I also like the idea of it getting blocked before it even gets into my network, instead if letting it in initially and then blocking afterwards - that’s kinda the whole job of a firewall after all ha ha
Why not just put everything behind a VPN and stop worrying?
How does this help with something like a mail server for a small org? Honest question.
It doesn’t.
It doesn’t, but I wouldn’t recommend selfhosting email for a small org. The low price of Office 365 or whatever Google is calling their business product now is far cheaper than the anguish of running your own server and dealing with spam, both incoming and making sure there’s none outgoing, and making sure your recipient servers aren’t considering your spam.
Our small mail server is doing OK. Incoming spam is an issue but not a massive problem. Outgoing spam doesn’t exist. Once a year the IP ends up on the Microsoft blocklist but using the deliverability form to submit mitigation requests is easy enough and takes half a day or so to sort out.
I’m looking forward to seeing what the Thunderbird team does with Stalwart.
That reminds me I’ve been meaning to spin up a server, install Stalwart and test it out.
If you’re running an email server for more than a handful of persistent users, I’d probably agree. However, there are self-host solutions that do a decent job of being ‘all-in-one’ (MailU, Mailcow, Docker-Mailserver) that can help perform a lot of input filtering.
If your small org just needs automation emails (summaries, password resets), it’s definitely feasible to do actually, as long as you have port 25 available in addition to 465, 587 and you can assign PTR records on reverse DNS. Optionally you should use a common TLD for your domain as it will be less likely to be flagged via SpamAssassin. MXToolbox and Mail-Tester together offer free services to help test the reliability of your email functionality.
VPNs are not a panacea by any stretch of the imagination. they are good for certain use cases but from OP’s description they would do next to nothing
It would protect all the services. Instead of having to secure each one, you only expose the VPN server and connect to that. You don’t have to worry about North Korean hackers breaching your services if they’re not exposed at all, only the single VPN service. Less attack surface, less worry.
And basically useless if you need external users to be able to connect to the services.
This is a scenario where a single node VPN would reduce, not increase OP’s security stance. You do have to worry about NK hackers breaching your services because they’re all exposed through the single node VPN server. Same attack surface, less knowledge needed to hit the target with the payload.
