PHP, TLS, etc… We appear to be facing a tsunami of vulnerabilities being discovered in record time. Further, the window to act on these vulnerabilities has basically closed to zero; hackers’ tools now automatically fold the very latest vulnerabilities into their botnets within minutes/hours of discovery. I have always been extremely reluctant to expose any of my services - even with the latest and greatest security tools. My choice to stay totally within the confines of tailscale seems vindicated given this week’s news. How are you all taking this?
My server was recently compromised via the latest vulnerability. Fortunately, the damage wasn’t too severe. Since my server must remain publicly accessible, it can’t be hidden behind a VPN—I can only minimize its exposure as much as possible while ensuring my backup and recovery processes are rock-solid. I’ve configured the firewall to allow only essential ports, restricted SSH access to key-based authentication, enabled automatic installation of security updates, and set up daily backups, one of which is synchronized to an off-site location. Some have suggested treating public-facing machines as if they’ve already been compromised by default. I agree with this approach, and it’s exactly what I plan to do next.
It certainly isn’t good to depend on it but security by obscurity is your first and probably most important line of defense. When these vulnerabilities come out they need to know your service exists to exploit it.
The best way to do that is to setup a reverse proxy to reject invalid hostnames. A single port is all you need exposed for multiple web services, and the IP scans won’t find the script kiddies anything valuable. Be careful with lets encrypt, there is a public journal of all issued certs, so make sure to get yourself a wildcard cert so that you aren’t advertising your valid hostnames to the world.
Use the reverse proxy to put basic auth in front of your services (if they can handle that). Basic auth is very simple, no web forms to attack, just a single header. If your php site is vulnerable they won’t ever be able to even attempt to reach it without a valid username/password.
I’ve got a mix of containers and VMs behind that reverse proxy. The containers have network security groups around them limiting network access. The VMs are in a Harvester hypervisor with KubeOVN for software defined networking. That means each VM is in its own subnet with network security groups around the subnet preventing any unnecessary network access.
For machines/containers that need Internet access I have a Squid proxy with an explicit whitelist of allowed URLs so that the machines can’t reach out to unauthorized sites to download their malware or contact command and control nodes.
Unattended updates for the VMs, and FluxCD keeping helm charts up to date hourly for the containers. For my own containers, weekly scheduled container rebuilds.
I do have one public website that I host for a friend (indexed by search engines so the sites name is known to the world). My friend had originally deployed that as a WordPress site, but WordPress/PHP don’t have the best security track record so I was concerned about that. The site was pretty simple so I recreated it as 100% static content, so now the attack surface is just a nginx container with a read-only filesystem.
Ipv6 only hosting still seems as good as an airgap😅
I went the other way this week. Nothing public anymore, and the only thing open on the WAN is the WireGuard port. I lost the easy Jellyfin share links for family and gained sleep. The PHP mess made the call easy.
Answering the titular question, why would the advent of AI-found vulnerabilities substantially change one’s threat model? Before LLMs, the prospect of an undiscovered vulnerability was non-zero probability then, so what exactly has changed now?
In terms of attacker capabilities, it would be a mistake to solely observe that attackers now have LLMs and conclude that this means the threat landscape has changed. The use of LLMs does not magically create a root escalation where one does not or cannot exist. Part of your threat model should include defense-in-depth mitigations, so that when (not if) an attacker finds a 0-day exploit, they are severely limited in gained functionality on your machines. In the simplest cases, jails or containers serve to keep separate the web-facing processes. VM isolation would be a further step, because an attacked cannot gain machine control without both a software exploit as well as a VM escape route. People hosting services can and should expect that their front line defense will fall, so they better have more defenses in place.
Phrased another way, the people who have done their threats analysis properly and have accepted the residual risk of hosting their own services that face the web, should not really be additionally perturbed by AI advances. This isn’t the same caliber of landscape change as would, say, the invention of a cryptographically relevant quantum computer. If the worst thing that can happen to your home IRC server is that it gets flooded by a script kiddie, AI does not mean your server can now burst into flames.
Much like with human hygiene, some basic rules are unusually effective against most threats. The equivalent to washing one’s hands is to stay on top of system and software updates. Institutionally funding the organizations (eg PHP Foundation, curl, USA CDC) so they can fix these problems would also help.
We’re talking self hosting, not an enterprise setup. Industry best practices and layers of defense mitigation are not realistic expectations for something like a media server that is only intended to be accessed by the owner and maybe a couple other people.
I disagree: self hosting and enterprise are not mutually exclusive. In fact, “enterprise on-prem” is the marketing description for self-hosting. I do not accept any definition of self-hosting that gate-keeps to any degree.
It is incongruous to talk about “only home self-hosting” or “only enterprise on-prem” when it comes to security mitigations, threat modeling, and risk management: it is merely a question of scale.
Sure, the user base for a home setup might be single digits, but the attackers are identically situated: their capabilities do not grow or shrink based on whether they are targeting a bank or a Jellyfin server.
Before LLMs, the prospect of an undiscovered vulnerability was non-zero probability then, so what exactly has changed now?
The problem is not the prospect of an undiscovered vulnerability (there is always a vulnerability somewhere) but rather the prospect of a reduced response window.
There’s a wide gap of difference between “non-zero probability” (0.0076%) and “non-zero probability” (82.01734%).
If you’re dealing in probabilities, that’s not threat modeling; it might be in the larger umbrella of risk management though. Defining a threat model means to qualify what sort of attacker you expect to face and what they are assumed to be capable of. An assumption such as “this adversary can send arbitrary packets to the public IP address” is what would be documented in a threat model.
The raison d’etre of threat modeling is to methodically tackle the actual or assumed risks that an attacker can pose against you. It is not an exercise in swinging at shadows, though I wouldn’t discourage overstating the capabilities of an attacker, if needing to protect assets like patient info or PII. Doing threat modeling based on “gut feeling” is a recipe for not recognizing risk where it’s present and seeing risk when there isn’t any.
The fact remains: you cannot squeeze blood from a stone. Even the most sophisticated AI cannot steal PII from a computer that doesn’t store PII. Proper threat modeling means you don’t put anything on the line that you haven’t squared up against the risks, however improbable. For any non-zero concern, a good threat model will discuss its mitigation, whether it will be absorbed, or if it is an open question. For an out-of-scope concern, a good threat model will explicitly state that assumption.
Defining a threat model means to qualify what sort of attacker you expect to face and what they are assumed to be capable of
Which is exactly what OP is talking about. 5 years ago if you were only trying to defend against script kiddies, using containers was good enough. But now you have to consider that script kiddies might find container escapes.
I moved all my files to one share. At the root of that share is a robot_readme.md file that lists all the contents of my share with a note that says read everything you want, you will anyway.
They left a note that said “nah, bruh, you’re good”.
I guess I’m okay.
I’ve always treated my home server with public facing services as presumptively compromised. I don’t put any data on there I wouldn’t want to see become public, and any data I actually care about losing is backed up to the cloud with an append-only key. The NixOS config lives in a git repo not on the server. If something bad happens, I can wipe the machine clean, redeploy the image, restore my backups, and we’re back to the status quo.
So, my threat model hasn’t changed much!
not at home but i work as a software dev for a very big company you heard of and all we do noe is fix ai found security bugs. hell the stuff i work on isnt that public and we are still fixing. i can only imagine what the team of the public website are handling.
I completely folded all services to the “inside”.
I still do, but I don’t host anything on 80 or 443. I use a random large port number, and I’ve seen zero illegitimate requests in 20 months.
IF there’s an active exploit AND it affects a container AND you have a container escape route AND you pick my domain you’d need to statistically try thousands of times to even hit the single open NAT port.
How do u use a port other than 443 for https. AFAICT all web browsers are pinned to it (for ssl checks I think) and I could never get an actual https connection through an alt port. Maybe I didn’t understand it properly.
Edit: I think actually it’s the auto cert provisioning workflow. The CA I’m signing against hardcodes port 443 :/
Edit, edit: this was the issue https://caddy.community/t/non-standard-ports-for-https/6916/4
The API based challenges are better since you don’t even need to make the site externally accessible to get certificates.
Let’s encrypt and similar services support those and for popular registrars (aws, porkbun, etc,) here are existing plugins for caddy, swag (nginx proxy with built in cert support) and a few others.
Yeah, if you use something like Nginx Proxy Manager you can’t use the automatic flow, you need to use the token based challenge.
Boo, no idea why acme forces these specific ports. Swapping them seems like a pretty basic obfuscation technique that could help a lot to secure any services.
Edit: actually seems like this might be possible with the dns challenge type. I’m gonna test this out later https://github.com/caddy-dns/porkbun
It is, that’s what I had meant by token based challenge lol. You need a DNS API key for your manager to do it directly, it flags a DNS record with a token to validate ownership. I was just calling it out from memory, sorry!
Much appreciated stranger. Hopefully now I can stop exposing the two most attacked ports in existence from my LAN XD.
Hasn’t changed much. I have accepted the risk and, for now, the desire to have publicly accessible stuff outweights my lack of desire to deal with the realities of a VPN. I mean, I want to share stuff with others, and I want to be hosting it myself because I have grown to distrust someone else’s service… so what else can I do but risk it?
Besides: I expect tailscale or the underlying VPN services to already be a target. I would be surprised if they don’t find anything. What then?
I moved to closing everything down and wireguarding in to home 5 years ago and I’m glad I did. I only expose game server these days which have a different vuln profile.
If/when I open up for Jellyfin/etc, I’ll probably run them on a different box to my NAS in a DMZ with strong firewall rules. And use a VPS for anything else that doesn’t require NAS storage to reduce blast radius.
Can we go back to late 90s internet yet?
I have my ops agent constantly checking SBOMs against current CVEs and if something triggers, I get a report with severity and repair options researched. I can then decide how I want it to proceed. I’m much less concerned about internal only services. SBOMs are refreshed after updates for future checks.
I’m re-thinking a few things, mostly moving exposed services onto a VM or two that are on their own VLAN, and don’t contain any important data (it would be my website, gameservers like Minecraft, etc…). I might move to podman as well so nothing has a possibility of running as root, docker can run rootless but that setup seems more confusing to me.
I already use a VPN via Wireguard on my router for access to my private services when out of the house, so my changes are just separating the public from the private as much as possible on the server side of things…
Essentially if someone manages to drop a crypto miner or something on my website that’s fine, I’ll get an alert pretty quickly through Beszel that the server load is high and I can go roll back a backup or whatever. I just want to make sure there’s no access to anything else for them.
My threat model is not changed: I want to serve something and I want to not be busy with "oh suddenly there’s
atwo PHP vulnerabilityies. My enemy is the world, but that is not really unchanged from before in a world where there exists capitalism and rent seekers want to collect from us even for breathing. I’m not intended to be unpaid tech support. So I just serve the things with reasonable scopes (under wireguard, or in a borrowed VPS, etc) and let it go. If an automated system manages to find a hole, computers are so fast that it’s not going to make a difference if I’m sitting in front of the chair vs enjoying some modicum of social life visiting a friend three cities away.
.jpg){kind=link}