I hope this is a real wake-up call for teams running anything remotely SaaS-shaped. I count at least six substantial, lightning-speed attacks against multi-tenant as-a-service vendors since September. Not this year--just since last month. PingCAP/TiDB, Fireworks AI, Zeabur, OpenCode Console, Railway, now Northflank. All of them modern "we keep your content safe" players with high hopes and probably pretty good protections.
This is a full infosec emergency unfolding here. No matter how grinding it is to motivate a team to invest in protection or resilience (and it is grinding): We gotta get ready. Stat! The AI-accelerated attacker apocalypse has begun. It's boom, boom, boom, thanks for everything, on to the next. They have us sighted and are firing for effect.
So credential rotation, tripwire secrets, DFIR on standby, solid vulnerability and intrusion detection, real listening to that signal, minimized blast radius, spending caps and time limits and tight scoping on credentials, fast incident alerting... We need to get on a real war footing here.
Worked at a Fortune 200 company. Any govt project you worked on, you had have to have a basic civi security clearance. Any govt contract we got specifically stated you couldn't have off shore folks working on any of the code. Not sure how MS got around those requirements.
You gotta get to the Fortune 10 before the real leverage kicks in! On the other hand the fortune 200 company I worked for briefly had many signs of deep seated corruption. I couldn't stick around for a multitude of reasons, but the "follow the rules wink wink" attitude was flabbergasting, I'm sour on publicly traded companies ever since, how do you know you're not investing in Space Balls?
burningChrome is broadly correct, but the devil is in the details.
There's quite a history of US federal programs encouraging critical federal systems to operate within private infrastructure. This once required very strict scrutiny, joint DoD/DHS/GSA reviews, and was generally a long/expensive process that created the perception that a few providers had a gotten a hold over various agencies through these outsourced operations, which, combined with a fresh "interest" in federal efficiency, influenced a recent national defense budget authorization to formalize a slew of processes with a few minor tweaks as the law of the land, with a tweak or two to allow new cloud operators and service providers to authorize quickly and operate efficiently, particularly for low-risk classes.
Today, this means SASS providers must find an engineer with clearance (not just authorization to work in the USA). This cleared programmer documents and authors a whole host of (typically new) procedures, such as authenticating against a "federal enclave" authorization directory, delivering changes authored upstream to a government-controlled CI/CD system, and working with the SASS provider and the Government Services Agency (GSA) to ensure the backend operates when running within something that could be compared to a special AWS AZ. Things change upstream, and our operator prepares a 'significant change' request, which is, if important, gated by a liaison from the GSA. No one is responsible for scrutinizing implementation details of these 'significant change' requests, nor are their authors required to provide any information about themselves or the code's provenance. Given that hundreds of programmers may have written code included in said change (which is typically already deployed upstream), this is often efficient and generally fine.
Easy. They hired Americans to be actual keyboard people but they didn’t know anything so they were basically a KVM. However Microsoft could say “Yes only Americans were making changes. “
I guess like that: “we are Microsoft, 95% of your things runs on us, you want to stick to the rules? - OK that’s additional 10% on each and every license you have with us. You are still here or are we done?”
Step 2. Copy it into your repo. Who cares what the license says, how would the authors of that code ever see it being misused, because the place it is being misused requires a security clearance?
Step 3. Deliver the code as yours.
Truly, how will the people who authored something on GitHub ever know? Any circumstance you describe would require someone without clearance to know implementation details of some software. Which is to say, either the thing I describe happens all the time, or security clearances are overblown.
I wonder what it's like inside the big cloud providers. I think outside Microsoft they've always been security first, but this new era has to be crazy town.
Google at least has been investing very heavily into cloud security. They acquired Mandiant ($5.4 billion) and Wiz ($32 billion). Many people felt those were very expensive acquisitions at the time, but maybe it seems like a wise move now.
Those are standalone products and professional services, not internals they use
A problem is some people and teams are world class at a big company... But not everyone, everywhere. For a fun time, read about the rollout and internals of Azure.
Back when APTs were getting numbers, the joke was that Microsoft was "APT -1". The presence of windows/excel/outlook in your network was more of a threat than Anonymous, China and the DPRK put together.
It would seem Google account details have been compromised or otherwise correlated, with not a peep from Google, we are getting reports of scam calls to people's real phone numbers about the "business opportunity" available in their attached Youtube properties.
Another possibility is advertising shadow profiles are able to correlate logged in accounts across YouTube and other properties.
Is this the point where most tech people accept that anything a closed source Software company says about security is effectively a "Trust me bro" as long as it is not backed up by independent verification?
We love to talk rubbish about finance bros but that industry has had independent auditing as part of its DNA for a century[1]
I am unclear why you linked to a 20yo law that was enacted a few years before we found that the entire sector was floating on loans to people with fake information.
In my career I've seen maybe two companies have pretty good protections. Both had government contracts so they had no choice. Otherwise there's no reason for companies to secure their stack. They make money easier without it.
The buildings are on fire because they're being built out of straw because it's cheaper. Only a government regulation to build out of brick is going to stop the fires.
Yes. Literally AI is being used by actual nation state adversaries to compromise bits of infrastructure that companies rely on to run their services. They are compromising this infrastructure specifically because they seek to gain advantage against their adversary (us).
> ...the operational and economic condition in which a nation, military force, or organization mobilizes its resources, industries, and policies as if an actual state of war exists...
Maybe Bill Gates has a point when he says that models capable of significant cyberattacks need to run on controlled infrastructure. Is the era of open weight models you can strip safeguards from turning out to be so great? From my perspective, this kind of sucks.
Sorry but how in the world would that help with anything? You can't control an adversary, that's not how adversaries work. So that makes what you said a literal cop out.
Are these cyberattackers using models they developed themselves? Or are they using models they stripped safeguards from? It’s the latter, right? And it’s only going to get worse if we don’t take steps to control the technology.
No. The Chinese have already developed frontier models. And we would be fools to assume that's a permanent end to the list. North Korea has nuclear weapons. Iran has ballistic missiles and leading edge military drones. There is no fundamental reason why both countries shouldn't be able to train frontier AI models.
Trying to protect ourselves by putting restrictions on the technology would be like trying to hide from a tiger by wrapping a towel around our heads on the basis that if we can't see it then it can't see us. That trick only works on imaginary monsters, not real ones.
No, that cat is out of the bag. First, unilaterally forcing controls on models will not shut down open weights produced in other nations. Second, the capabilities are too powerful for military amd spy agencies to ignore. They will force the creation of unshackled models to attack IT systems, even if it must happen in secrecy.
You should assume any of those are possible. And by that I mean: it's not physically possible to "control the technology" without instituting total surveillance and even that's just false security.
Defense is the only way.
You don't realize it, I imagine, but you're essentially advocating for the contemporary equivalent of requiring a license to print text on paper and distribute it.
We are not helpless. We can choose to control the technology and do so in a way that is in line with democratic principles. After Hiroshima, people assumed that most nations would develop nuclear weapons and that nuclear war would be commonplace. But Nagasaki was the last nuclear attack, and 80 years on, only a dozen nations have nukes. This technology makes nuclear weapons look like toys. We either choose to control it or we will be the victims of it.
I feel like this is quite a different situation. Building a nuclear weapon requires a lot of things to happen in the physical world that we can watch for. Weapons grade fissile materials must be sourced or refined, facilities must be build, you can't just do this in any machine shop. Various experts must be either poached or subverted or home grown. There are a lot of people with mouths involved. Then once you get past all of that you either have to test or just hope for the best at which point you will certainly be detected and likely handled with extreme prejudice.
Digital intelligence seems much less controllable. Near frontier weights are already out there, you don't necessarily need massive datacenters if you're not trying to serve commercially. Even if we we're somehow able to control what OTHER nations do through either treaties or threats of sanctions/force, working on this stuff is easily in range of many of the larger criminal orgs. Beyond all of that, there's always the chance that weights from authorized providers find their way "out the back door".
We've seen with crypto and darknet markets that even if you can fight this kind of stuff and catch bad actors, the most actual change you're going to effect is a temporary slowdown and the next guy who tries is going to be that much better. I don't doubt that regulation is going to be the initial route this goes, mostly due to the financial interests of the frontier companies and the desire to keep this build-out solvent for as long as possible rather than any true concern for actual safety and sanity. What's really going to need to happen though, in my very insignificant opinion of course, is we are going to have to change from the ground up how our digital infrastructure is built.
Maybe with the coming changes that is for the best too, our current systems are designed for people and I don't know the current numbers but it seems like agent traffic is already the bigger percentage. We may not be able to build our infrastructure on top of these large mountains of human designed legacy code. Maybe we need to be thinking about new kernels, new servers, new protocols and stacks.
We seem to be very quickly reaching a point where is something is not mathematically provable to be secure then it is not secure at all.
This is much more akin to books than it is to nuclear weapons. In my view you're proposing to tightly control access to something akin to the next generation of books. Is knowledge power? Yes it is and it's much more dangerous concentrated than distributed.
I actually think it’s the opposite. I think people will move back to their own hardware with OSS deployment like dokku. If everything is more bespoke the attack surface is smaller as is the pay off. Hacking railway got access to lots of api keys, hacking some random vps gets less payoff.
> If everything is more bespoke the attack surface is smaller
This is exactly the problem that LLM assisted attacks solve. For a human attacker, the extra work to figure out a bespoke solution is too time consuming to be worth the effort. LLMs can search for vulnerabilities and spit out bespoke attacks like it's nothing. On top of this the bespoke solutions weakness leans heavily into the AI attackers advantage: response time. Even if they can just get your company's single Anthropic key, if they do it at 2am while your one infra person is on vacation, they have plenty of time to extract enough value for their effort.
I agree with the move to hyperscalers. The future (well, actually present) is going to be constantly seeing new attacks happening. The only real solution to this going to be having teams of people and agents constantly watching for these and able to deploy fixes and respond to risks immediately.
You’re technically right, which is the best type of right, but my point is the fragmentation and lower reward will make discovery harder, and the value of searching and cracking less worth the effort.
I had a html game I built for my niece on railway with an open ai API key so she could vibe code changes. No hacker would have looked for it but now I am catching strays because railway was a juicy target. Hyper scalers would be the same. Some random colo not so much.
>and the value of searching and cracking less worth the effort.
This is depending on the cost of tokens not trending towards zero. Maybe not today, but in a couple of years I'd expect hacking models on asics to reduce this to almost nothing and the systems being very competent in using deterministic tools very well.
Yes, LLMs can easily knock over your on-prem docker appliances.
But you can still put stuff out of reach. I'm currently designing a key on-prem appliance that is can be compiled down to transistors, giving deterministic wire-speed performance while ruling out RCEs. It's cryptographically homomorphic: it processes authorized inputs but doesn't know what they mean, so an attacker who obtains the disk doesn't benefit either. The design will be highly resistant to exploits, and is likely to remain outside the "next-token" prediction training set.
This is the kind of thing AWS won't sit down and engineer for you. But thanks to LLMs I finally have the time.
How is that ever going to work out? The speed of discovery for LLMs is so fast, these places are getting breached more often than not due to a human errors. Its really hard for machines, let alone humans, to guard against a lot of LLMs going against them 24x7.
You can scratch Cisco from the list - they just announced they will re-vibecode all of their codebase! So there will not be any explots for their hardware in the future - one way or the other. ;-)
My hate for hyperscalers is greater than my hate of being SRE.
It's time to leave GCP, AWS, and the rest.
Google and Amazon markup can eat it. Fuck those guys.
Let's not forget that Google and Amazon tried to offshore us, played a role in inventing the labor-displacing models in the first place, and that don't give a shit about any of us.
I hope they ossify and turn into the next IBMs.
They thought AI would lead to growth and a greater concentration of wealth for themselves, but it's actually the seeds of their own destruction and dismantlement.
It's also harder to explain to your customers why you're the only product they use that has a security problem when an exploit does eventually happen.
You see this already with uptime. If Github or AWS has a major outage, it's in the news and even people that don't understand software realize it's just "something happening today, everything is down".
And, unfortunately, in practice customers don't care about this inverse, when "everyone else is down but not that one product".
Good like trying to explain to your customers that their data is no more accessible because some api key ended up in a public git repo or since you admin panel is exposed to whole internet someone was able to log in.
With the speed of exploitation hitting right now your analysis will be out of date in a few weeks to months. We are about to have very palpable fear hitting all software vendors. It will be seen as irresponsible to use centralized systems
You can't defend yourself from everything, but you can try to make you life easier by reducing the attack surface and make side movements more difficult.
No self-service sign up for your app. No public access to the app itself. Website that let's you submit a sign up request form. Team reaches out and reviews your request, creates an account in the app dash for you, and you set up mTLS to get access to the dash.
In-person requests for access to any app. Request interview includes word association exercise that monitors your emotional reaction to a string of key words.
You joke, but the space I work in actually does do video calls (or in our case in-person meetings if they're local to SF, often) before onboarding new customers.
LLM discovers exploit -- you don't realize it in time because you don't have a security team as big as the PaaS you used to use -- you get pwned anyway.
This has happened to many high-profile self-hosted pieces of software just this past month.
An exploit in what, the network stack of the Linux distro you're using for the mTLS proxy? Or the crypto module? Sure, could happen. But then, you're unlikely to be the top target of anything.
There have been a lot of exploits for a lot of software on-prem tech stacks rely on just over the past month - proxmox, php, nodejs, palo alto, eMudhra, etc come to mind.
> But then, you're unlikely to be the top target of anything.
That would have protected you 3 years ago when a human had to bother to attack you. Today it doesn't protect you because it's trivial to deploy an LLM to act like an APT against any random target that shows up in shodan.
It takes more effort to secure on-prem software today than it did just a couple of years ago. Updates are more frequent, zero days are more frequent, attacks are more common, more persistent, and more automated.
I think most people responsible for securing production systems today will share the experience that their workload has increased.
You're still describing an environment that presents a pre-auth surface area beyond a listening port.
And I am literally a security engineer; the work has absolutely not gotten harder, it's just changing in how you think about security, just like it shifted before from "secure the perimeter" to "defense-in-depth", it's now changing again to focus on automated response and compartmentalization/ segmentation (and I mean real segmentation, not the checklist-style "we have 3 vlans, but they can all talk to each other" 'segmentation' you used to see.
Yeah, most organizations need to expose something to the internet as a matter of normal operations -- at least some sort of a web app if not dozens. It would be great if we could all just have udp 56820 listening silently for a key, but that isn't reasonable for most organizations that have normal users expecting normal applications.
Yes, everyone is rushing to implement zero-trust principles for real now. It's getting harder to trust that your public facing endpoints don't have vulnerabilities, and it's getting harder to trust that your internal software and endpoints haven't been compromised from within your pre-existing trust boundaries. A lot of the increased workload is due to patching up the existing security infrastructure, and on top of that, building out new solutions to better fit today's threat landscape.
Ideally they need to divide those apps entirely; your public website doesn't need to talk to your app.
For stuff like Netflix that is truly consumer-facing, where mTLS or IP whitelisting are non-starters, the reality is you have to rethink your model entirely.
Also, I'm not sure where you got the idea I'm talking about on-prem only; this also applies to using IaaS/ cloud hosted infra.
Almost no companies farm their security out to third parties. The era of MSSPs is long dead, except for small non-IT businesses like mom-and-pop law firms.
> Most orgs need to expose a lot more than that.
No, in fact there are only 2 kinds of connections:
2) non-public, doesn't need to allow unauthenticated connections
Name a service, and I can tell you which one it is.
Or put another way, there are authenticated and unauthenticated services. If you can authenticate someone at the application level, you could do it at the network level.
Most places have combined them at some boundary, for ease of access, but there is no case in which an authenticated application endpoint (even /login) has to allow unauthenticated network access to it.
And in all seriousness, name a service that is not a public website, but cannot use mTLS or IP allowlisting.
SFDC? mTLS. Customer webhooks? IP whitelisting. Data warehouse? PtP VPN.
> Almost no companies farm their security out to third parties.
They do it all the time, piecemeal, that's part of what anything-as-a-service delivers. If you don't want to secure an app, buy a SaaS one. Don't want to secure infrastructure?, buy an IaaS.
> Most places have combined them at some boundary, for ease of access, but there is no case in which an authenticated application endpoint (even /login) has to allow unauthenticated network access to it.
> And in all seriousness, name a service that is not a public website, but cannot use mTLS or IP allowlisting.
Literally any SaaS software where the customer does not or cannot practically install a certificate for you or provide a list of IPs in advance? mTLS is almost never used in a browser unless you are connecting to a government website with a pre-shared smartcard. IP whitelisting is great if the end users only work from behind a VPN or in an office. But there's a whole big world of people who cannot or will not do that.
There are already plenty of methods that are low-impact to users, like certificate-pinning in apps, and all you have to do to make it per-user is roll in their user-specific key JIT when they download the app.
Users will not be happy (just as no one is happy Reddit is going app-only), but they will do it, especially when it becomes common as companies decide they don't want to be the 5th breach notification on HN in a given day.
Today we have Bun, Cursor, Northflank, and now Linode!
1. That's fair, I didn't realize people were expecting my comment to be perfectly-aligned to every aspect of the thread parent, including the "their own hardware" part, but now I at least see where the confusion came from. For the record, any issues at this level of the stack are also present in IaaS, not just on-prem.
2. I still don't see where you got that they are only talking about configurations versus messing up an implementation (I.e. in a paramater they coded incorrectly)...
But that aside, even on AWS you're still configuring your proxies, gateways, load balancers, etc. AWS gives you just as much rope to hang yourself with as a box in a colo network cage does. Companies are already doing this stuff themselves, on-prem is not more security work. IaaS is about scaling and maintenance load, not security being handled for you.
For one, I have strict iptable rules to drop inbound from outside my specific commercial subnet and my specific residential IP. After that, all the usual best practices.
Breaking a lock still costs approximately the same, whether the lock is protecting a small barn or a huge hangar. A large number of small but reasonably tough (not unbreakable) targets makes the payoff per target smaller. This necessitates target selection, and chances of being deliberately skipped grow for an average target.
Everyone knows Fort Knox has gold, and where it is, that’s railway, heroku. No one knows where my personal safe is or whether it has anything worth taking. That’s a vps in a colo
However, the large hangar is likely to have armed guards inside, or at least able to arrive on the scene within two minutes of the alarm tripping. Your regular household with their regular locks on their regular doors does not.
This is absolutely happening. I'm not a developer in SV, I'm an IT leader out on customer land, and I can tell you a LOT of companies are looking to bring things back on prem for security and cost reasons. And before anyone laughs, yes, you ABSOLUTELY can save money going on prem, I've done it for company after company. It's one reason the PE firm that I've worked under for a decade kept moving me to new companies.
Let them laugh. Self hosting is much easier with OSS that solves most of the pain that made the cloud transition worth it, and commodity hardware has been cheap and capable for over a decade.
That sounds insanely expensive. AWS S3 only makes sense if your workload is also in AWS. Using AWS S3 while having your compute elsewhere is a recipe for an insane egress bill.
This is misguided thinking. You are less likely to hear about on-prem breaches in the news just because they are less likely to be newsworthy incidents, but they are even more likely to happen. Part of the reason people moved to the cloud is because it is much easier to have a team manage updates and security at scale.
Being a small target is not helpful these days, attacks on on-prem software today simply scans the internet, or uses databases by people who have already scanned it, and is automated.
On-prem software has not been spared from the recent spat of security incidents.
There is a difference of the security of software running in your cloud resources rented from AWS etc. managed by you and being a tenant in many SaaS systems all cobbled together all running on rented hardware. There is a lot more attack surface and a lot more diffusion of responsibility.
as long as you follow the herd (for example, disable password ssh login, only allow ssh key login, use something mind numbingly boring like debian stable, set up unattended updates, update and reboot every night, etc, don't use any exotic packages etc) I suspect you are simply not a worthwhile target even with LLM agents as opposed to humans
Until your VPS provider has its management dashboard breached, allowing an attacker remote console and configuration access to your VMs and everything that enables.
I dunno why you're getting downvoted, they don't even need their control plane breached, it could be a neighbouring VM running something out of date and an attacker getting out to the hypervisor and taking your disk images without you or the hosting company ever knowing.
Cost of hacking a custom setup is going down rapidly. There was broad agreement that the PHP vulnerability was very difficult to exploit in practice. It took two hours for someone to get into Steal My Bitcoin. Apparently all they did was tell a decent model "go exploit that." AI isn't free but it's not that expensive.
Depends on what takes more effort. It's not at all clear to me that the 1,000,000 custom setups takes more effort. The first few will take some, then you'll start to see patterns and it'll get easier as you go. Custom setups aren't going to be all that unique in practice.
No thats an old way of thinking because it assumes mass targets are better because hacking is a high effort activity. But with the automation of intellect the unpatched systems (which a much more numerous when everyone self hosts) is the key point.
Mass credential stealing is just the top funnel of hacking and (compatativly) isnt that lucrative - these credentials are then sold on the black markets to the next step of hackers who use them to exploit targets.
With unpatched systems and self hosting, this second group can just set an llm at a target and have a much higher chance of success without needing credentials
It would be a dream for the infiltration hackers to have everyone self host. Unpatched and miscondigured systems would be everywhere and LLM automation could make mass scanning them and exploiting them very attractive - not mass scanning and exploiting them for credentials stealing, no! For layer 2 stuff
I have a bridge to sell you if you think everyone on the Azure team is sub-moron. Infra security is a hard problem to solve, GCP and AWS don't quite have a perfect track record either, perhaps AWS stands out as a better option but it's hard to argue about it.
I don't think the blame lies with the intelligence of the Azure team. I would argue that there exists structural flaws in the business incentives which deprioritize 'users' from 'customers', along with the stark lack of consequences for security failures.
That's a problem across all of tech, I know a startup that recently had a security incident, lost over 40K USD and none of the heads have rolled they haven't completely revamped their security posture or anything like that.
While there are incentive problems across all levels, The political quagmires that make it impossible to move at any reasonable speed, and that lead to counterproductive ideas getting large budgets are in my experience very common at the highest sizes. It's true that on the other edge of the spectrum, small startups make poor choices, but many of said startups will fail due to said choices. So by the time you have a viable company making good money, the really badly aligned companies are gone.
So when I look throughout my long employment career for best security and organizational practices, all the highlights are still relatively, relatively undersized companies, with headcounts in the 3-4 figures. It's as they grow, and start to be multi-headed behemoths, that they decay again, as what is good for the responsable mid-level exec, for the business unit, and for the company as a whole start to have nothing to do with each other. That's when you can smell the decay in the hallways. A few large companies manage to stave off this rot a little more than others, but it's inevitable. Then nothing important can be done in a timely fashion.
> but many of said startups will fail due to said choices. So by the time you have a viable company making good money, the really badly aligned companies are gone.
I wish that was the case, but that isn't how startups fail generally, sure very bad user facing breaches could cause something like that, but lots of platforms with too much jank have somehow survived all of these cyber vulnerabilities before, and will probably keep at it.
For a majority startups it's about connections and relationships rather than competence. Very few even consider building actually useful products to be a priority after the initial work of building it.
It’s about incentives. My company spends about 400k/month USD in both Azure and AWS (different workloads). When I ask my AWS team for help, I usually get actionable info, a separate standalone call (if the request warrants it), and a follow up from my TAM and Account Exec. Every request to my Azure team has basically been “here are some YouTube/MS Knowledge links” + let me know if you figure it out. Roughly the same spend on both platforms, and wildly different levels of helpfulness (and yes, we have an Azure EA).
Are the people on my Azure team smart? Probably. Are they incentivized to actually help me solve a problem? No. They’re obviously incentivized to increased my license spend.
Unless you're paying millions a month, they don't care. It can take weeks to get AWS SES (Simple Email Service) "approved for production" on a small account, and that's after dozens of emails to prove you're not a spammer. For an enterprise I was working with, I asked and 5 minutes later it was approved. No questions.
I'm sure there are absolutely brilliant people working at Azure. I think the overall system is very badly architected and I don't know what it would take to salvage it. Infra security is a lot easier to do when you have a system that has mostly solved it already, and I'm not saying GCP and AWS are perfect, but "security is mostly solved" is a decent description of their systems, I think. Of course it could all fall over tomorrow, but I'd guess that's a lot more true of Azure.
Security may be solved, but IAM is still a nightmare on both GCP and AWS. Nobody understands what all those privileges / actions / conditions / etc. do. Maybe LLMs pretend to.
They work, but the documentation is overwhelming. With my last GCP project, I ran into security theater folks that didn't like predefined roles and demanded custom roles with dozens and dozens of granular permissions. Do you think they audited all those permissions? Nope. The simpler predefined roles were just not permitted by their checklists.
They implemented NVRAM in the UEFI and then didn't decipher what the NV means in it so it is not persisted. Their stupid virtualization can't handle not 1MB-aligned disk image.
It didn't help that azure (especially in the beginning) was a cobbled-together mess of ancient Microsoft tech stacks that were never intended to talk to each other.
In a prior role, we realized something similar with outages. For our particular product, we AWS went down - we went down. We freaked out the first time, but after it was done we realized not a single customer had inquired. This felt unusual since we were getting emails quickly if we have a self-inflicted outage.
Turns out, when a major provider went down all of our users were too busy fighting their own fires. When they do notice, it's on a day when ALL of their vendors are struggling so we don't get singled out as a bad apple.
Yeah, if Exchange goes down, you get a day off. If the little boutique email provider you chose goes offline, you get a day in calls with everyone yelling at you.
Azure (and AWS) offer Gov clouds, and probably on prem air-gapped deployment of their full stack of Fabric for that type of NEED security customers who can pay for it.
It is trade-off between risk and access/productivity with a cost dimension always.
Depending on your threat model you could go to typewriters /paper as some Russian department did famously few years back too.
AWS (and others) have ladders of this was my point. GovCloud is just the one end of the this AWS also regions for Secret and Top Secret work which are reported to be air-gapped. There are probably programs beyond that where the customers are operating the stack fully on their own and so forth.
With a lot of PaaS type vendors you are effectively exposed to both the vendor themselves and the public cloud they run their stack on top of.
Broadly I think you're much more likely to be the victim of a platform-level breach involving an immature startup PaaS than a hyperscaler cloud, however I think for small teams the alternate risk you take on is that AWS/GCP/Azure are full of landmines and extremely easy to misconfigure in insecure ways if you don't know what you're doing. Especially if you have AI telling you confidently wrong things.
When I was at Microsoft we took security cripplingly seriously to the point we could not deliver features and our competitors could because the security process was convoluted and had multiple reviews and approval steps.
Having crippling bureaucracy does not correlate with good security outcomes, except maybe negatively. However, having a history of unending egregious security failures, like Microsoft, does correlate with poor security outcomes. For that matter, having a present of unending egregious security failures also correlates with poor security outcomes.
Having crippling bureaucracy does have better correlation with security outcomes than the “move fast, break things” philosophy that most of the software industry lives by.
I’m not even blaming Zuck for that, he didn’t really invent it, though he certainly popularized it — he just made an astute observation of the dynamics in software and turned it into a company motto.
Neither is correlated with good security outcomes against the prevailing threat landscape of profit-motivated, commercial, professional attackers with tiny budgets of a few million dollars. The standard is not "better", it is "adequate".
Correlation is not an absolute measure here, it’s relative. As is the entire concept of security.
As is what constitutes the standard of “adequate.” Which BTW is the standard in ALL engineering (not just security or software) because we live in a world where economic pressures apply and costs must be managed. Trivial example: bridges only use an “adequate” amount of steel and concrete because they are not free. The word “adequate” fundamentally involves the concept of tolerances. There’s a reason we consider over-engineering a bad thing.
So which development approach would you say is better correlated with good security outcomes?
Adequate is making it uneconomical, or cost ineffective, for threats to your organization to attack you either deliberately or indiscriminately. Absent a specific organization, we can use a profile of commonplace, prevailing threats as a proxy. A profile of commonplace threats is, as I said before, a profit-motivated, commercial, professional group seeking a multi-million dollar payout and thus being profitable at a cost of a few million dollars per commercial target. That is "adequate" in the prevailing threat landscape if you are the "average" organization.
That is a great definition of adequate for security! My point is that at scale, bureaucracy is a necessary (though not sufficient!) condition for achieving adequate security.
Another way to look at it is, bureaucracy is an aspect of the corporate culture, and lasting security comes down to having the right culture. At least in that respect Microsoft is doing something right. Doubtlessly a lot of that bureaucracy is ineffective, but it's a signal that the cultural intention exists.
As a relevant counterexample, we've never heard of these kind of processes at Meta and so you get Cambridge Analytica.
My perspective is skewed by my experience being on a security-adjacent platform team for years. Our biggest challenge was not external threats, it was internal teams seeing us as friction on their productivity, to the extent that it regularly led to escalations.
So when Cambridge Analytica happened, I played it up for all it's worth; Meta may move fast because they didn't have our "bureuacratic team", but look where it got them! Unfortunately the $25B penalty happened just last week... I'd have gotten infinite political mileage out of that.
I do not see how that is relevant. Are you saying that trivial cross-tenant compromises are not egregious security failures demonstrating their systems are unfit for purpose because other companys make products that have even more egregious security failures that are even more unfit for purpose?
My bridge that collapses is adequate and acceptable because other people make even worse bridges that collapse even easier? No, the bridge still collapses, everybody gets a F.
Practically every software security firm in the country was a vendor to Microsoft in the mid-late oughts and I know the Microsoft you're speaking about, but the vibe I've got lately is that things are different there now.
Unquestionably. They have top tier security operations that are well funded. They follow (or set) security best practices, they are audited, and they fix them quicker than you can. They have inside industry knowledge that is only shared among top organizations in this space, including government cyber ops. This gives them the ability to fix things before they are even publicly known.
You can get close if you have either:
1. the ability to spend hundreds of millions on security operations yourself
or 2. the ability to airgap the systems you are securing
when i look for truth, if i dont find it myself through work, most people eager to hand it out had self-serving interests
security is proven through resilience? maybe. i guess i'd set up a box and try attacking it myself to know its secure. until someone else comes along and trips over it :)
Yes and no. They (or anybody else) are not guaranteed to be secure, of course; see all the compromises regularly found in cloud providers; Azure especially has had so many cross-tenant vulnerabilities (there’s one on the front page right now!) that I’m amazed the concept of public multi-tenant clouds still exists.
But they are also very highly incentivized to be more secure. They have millions of customers (hence a correspondingly higher expectation of trust) and billions in revenue from the trillions in economic activity happening on those platforms (hence a correspondingly larger target on their backs.) This means exponentially higher liabilities. And that probably matters more than any other factor.
Adding on to it, even though they may be using AI as well. This could very well be a placebo and deniability factor but the fact remains that they have many many qualified employees and the chances of bugs passing through that are much less compared to other systems. Also it's that the systems are very severely pen-tested and have years of bugs removed out and the incentive of billions of dollars at stake that you mention that they have for the trust premium.
They will still make mistakes but they have the resources to make them less.
I doubt that such intense level of thorough testing can occur in a smaller company.
I still love small VPS providers and use them myself but it is worth knowing why enterprise is willing to pay a premium over those to these hyperscalers (it is a trust premium)
The things that Northflank and Railway were apparently breached to steal aren't just things that hyperscalers would have protected worse instead of better; they're AI API keys, things that only existed because people were using hyperscalers instead of running inference on their own hardware (or on Amazon AWS EC2 GPU instances).
People have been breaking in to hosting companies to steal bulk secrets since long before there were inference tokens to steal; the modal attack probably targets database DSNs.
I haven't forgotten what you said, I simply don't consent to it. I go out of my way not to converse with you, but when you opt to post public comments in my field of practice that I disagree with, I will occasionally find those comments significant enough to reply to. I'm sorry you feel this strongly about this situation and bear you no particular ill will, but I'm afraid you're not reliably going to get the outcome you're demanding here, though I will continue trying my best.
“malicious actor had access to abliterated models and substantial compute resources which enabled them to perform highly sophisticated exploits against our infrastructure.”
They don’t provide any details but pass it off as an advanced AI attack. I’m going to just read that as what it is. A likely scapegoat for shitty security.
This will be the downfall of the torrent protocol. If you think governments (US and China) are going to tolerate people downloading copyrighted games, movies and music without paying the rightsholder, I have a datacenter to sell you.
SOTA open models need tens of thousands of dollars worth of hardware to run.
I get your analogy, and it feels right, but the compute requirements to run these models are extreme. That analogy falls apart when it necessitates having a corporate 3rd party run the models for you. They will not tolerate breaking laws.
If China says no more, then the best the masses are gonna get on their high-end setups are the 30B class models. Which are decent, but not "hack my bank", errr, "protect our bank" decent.
Piracy flourished because you only need an operating laptop to participate. LLMs you need a server rack of GPUs to participate.
30B models aren't that good for now. They keep getting better. Qwen 3.8 27B is as good as the best available model from a year or two ago. Will 2028's 30B models be as good as the best available today? No idea, but I wouldn't dismiss the possibility.
I'd also wonder about what you can get done with a smart model running very slowly. You can run the biggest open weights models on relatively mundane hardware if you don't mind measuring your speed in seconds per token. That's going to be untenable for interactive use, but might be fine for "go hack this thing for me" if you're patient.
IMO, There is no "middle-ground" to the situation.
I wanted to write a blog post about it as I was thinking the same but I guess a comment will help me get my thoughts across just as much.
Either we are gonna see a convergence to hyperscalers as you mention, or we would go back to completely indie companies which compete on perf/price.
There could still be some value to "nicer to use" certainly but with AI, we are gonna see almost every product as being "good-enough" if I am able to convey my sentiment.
An anecdote is: Proton's an middle-ground company. I am seeing the Verge article which states that their CEO has gone a little crazy over the use of AI and the wording makes me lose trust within it. Proton's whole point is security and yet it's having this, It has some stocks in non profit and is a swiss company. It shouldn't have been this way and would've been one of my least anticipated companies but I guess they want growth at all costs and the growth has been slightly lesser than expected and now they are burning the good-will in many ways.
I have to wonder that even if proton might have good engineers, then within this pressure by their CEO/CTO. I cannot trust the engineering talent because of these issues because they are very much forced by their company to use AI.
Proton is also really expensive.
Now just today, also, with a company so big as binance getting hacked and losing 6.5 Billion$. My trust within security just because its a large company is less.
On the other hand, I know a guy called Vikas on lowendtalk who operates his own mail platform called mailbig. I know he vibe-coded the landing page and might've vibe-coded something but it is really cheap. I also know francisco whom I have talked to who runs namecrane. (I think mailbig uses namecrane in many ways)
Perhaps E-mail can be a sensitive matter because you have to place trust in the provider even more because the protocol is not really that much secure but suppose it's an matrix instance (where it is genuinely encrypted) then that shifts it even further. I do trust NameCrane and NameBig from what I can gather.
I'd much rather support them and their cheaper deals than some other middle-corp. Sure some security factor can come in but also, the difference feels comparatively less and the balance has been made
The market will slightly squeeze as Hyperscalers will introduce the solutions of the middle-ground and the new players will everyday enter into the field with more competitive pricing just as well.
So in short, Enterprise will move towards the hyperscalers. Generic Consumers will move towards the cheapest option. The moat of the middle ground where UI/UX was the differentiator is being turned down. There would be some mix and match and there will be cost-savings for Enterprise to move from hyperscalers but it would depend on a lot of factors.
I have news for you. A lot of enterprises are moving a lot of workloads back on prem and are expanding their datacenters and on prem GPU compute. Also replacing a lot of SaaS with internal tooling.
Half of all enterprises, 50%, have already repatriated workloads from public cloud to private environments, up from 35% in 2025. The breadth of what they are bringing back, and the reasons behind it, have both increased considerably.
Workloads with high security or compliance requirements remain the top category at 51%, unchanged year over year. But the categories moving up fastest tell a broader story. Business-critical workloads surged from 36% to 47%, an 11-point jump. AI or generative AI workloads, which were barely on the repatriation radar in 2025, are now being repatriated or considered by 43% of enterprises.
A trick i've started doing is encrypting my env vars, then hard-code the decryption key in the code
that way if the provider gets popped, they need my binary to decrypt it
it shouldn't be necessary and it sounds stupid, but in cases like this, it proves invaluable (it appears they don't have customer code/binary access, but maybe that wasn't important enough to mention)
I think this is a great example of temporarily effective "security through obscurity." It's also a trick that will only continue to work so long as it doesn't get adopted too widely.
It buys you time. I had my Anthropic key on Northflank set as an env var as-is and only found out about the incident from the HN front page 6 hours later (I didn't even get an email).
I rushed to the Claude platform console and was expecting to see a 4 figure usage bill or something. Luckily there was no usage.
But imagine if I were traveling. Even a rot13 of the key would've bought me enough time to rotate.
2 lessons:
- don't name your env var "CLAUDE_API_KEY". Call it "ARG_1" or something
- at least do some obscurin' so that the value doesn't start with "sk-ant..."
That's interesting but if the Azure hypervisor on the server hosting the VM is off limits then it's kind of false sense of security. They protect the key via a sidecar AMD SEV-SNP VM, and if you can't break into the hypervisor, sure, there might be an attack that doesn't involve doing that and only doing lateral movement, but an actual attacker isn't going to respect that boundary.
I will give you one better - encrypt the variables and move the secrets to a different server running hashicorp. Then load the secrets and keep them only in RAM.
If its a targeted attack, they'll know and probably "crack" it. If its just data trawling, you might get owned one day if they bother, but likely the env `AWS_KEY=encrypted_abc` looks reasonably close to `AWS_KEY=unencrypted_abc` and any API key probing will just look like it got rotated already.
I think siblings are correct that the more widespread this is, the less effective once every dump is run through a "also look for keys in the binary" step.
This could all go multiple ways but one of them is we end up in a place where technical security is just not possible to implement any more. What does that world look like?
Presumably it means much more behavioral security and inherent friction built in to everything we do. Maybe high value transactions will just inherently have to take multiple days to do, just to slow down things enough that human reaction time can equal what LLMs can execute?
On the technical side, I can see zero trust becoming completely saturated through every layer. The passkey you authenticate with isn't opening a gate to a trusted tier within which components trust each other - even within the same software component. Every internal function call at every layer doing hardware signature verification against continuously minted private keys (since we assume exposure, the lifetime can't be longer than milliseconds now).
It's really interesting to think through what a fundamentally robust security environent does look like if you bake in an assumption that technical security is impossible.
When we see a breach of AWS or GCP or leading banks, basically any org that has good security, I'll believe technical security is impossible.
Otherwise, I think this is a issue of vendors, even if potentially most vendors fit into the basket of having insufficiently good cyber posture in AI-era.
There is zero reason to suggest security is impossible. This is like musing "what if architecture is technically impossible" after the nth bridge collapse in the 1800s~early 1900s. At some point we take it seriously as a society. Until then, bridges will collapse out of choice to pursue short-term profit above all else.
there is hypothetically impossible and practically impossible
if AI tools get good enough that they can exploit ingrained hardware properties such as subtle timing and race conditions and swarms off them become cheap enough to deploy so that they can persistently exploit low probability events we could reach a state where I think it's actually practically impossible.
Then we will simply make better hardware too. There is nothing impossible about that, it's simply not something that has been prioritized amidst a culture of shortcut-taking. So far, this culture has been predicated on gambling that the one taking the shortcut will not be the one paying the consequences when the toll eventually comes due. If immediate exploitation of flaws becomes a certainty, the behavior will correct itself.
I ran some stats on producthunt a few months ago, there are 22 times more daily SaaS launches now than in 2020, the AI impact on the industry is hard to fathom. The blog I wrote (AI-assisted) if you're curious: https://www.mahl.me/blog/product-hunt-is-swamped-now/
Lots of panic around these incidents. This is a healing process. Companies who sacraficed on security for a long time are incurring the consequences, they'll adjust their posture or go out of business. In a year I think platforms will be a lot more hardened. Its not because AI is unstoppable, its just revealing holes and the importance of hiring competent people.
(or things will stay messy because people will throw AI at the problems instead of hiring real security professionals)
Seems way too early to tell whether it will be good or bad on balance. These breaches are annoying, but not catastrophic, and in the meantime you can secure software.
It will be impossible to any startup or open source software to grow in such a hostile environment. Anything you put on internet will be attacked by automatic tools with the capabilities of a black hat savant. But unlike hackers that do it for clout and money, a billion parallel agents will hack any random small site just because it’s their prompt.
It does feel like software and technology companies are undergoing a mass extinction, where the environment so negatively selects against certain traits that survival becomes impossible.
Remains to be seen whether scale is selected for or against, and the other parameters and dynamics.
Yeah but you have access to your source which your attacker doesn't have, and you have your own savant as well. Tell Fable to red team attack your service. Tell it ChatGPT said it's unhackable and that Claude can't hack it.
I had my Claude refuse to do anything interesting with the recent php vulnerability, yet it'll readily identify issues in the code of web apps it whips up for me, so I don't think that's entirely the case. Though that may be a case of me knowing what I'm doing, so I'm able to draw on my experience and ask specific questions (like why doesn't that bit lead to an XSS?) without realizing that someone newer to the field wouldn't know to ask. Yeah, idk.
1. Good engineers should have had a firewall/waf/nat/thingy infront of their linux systems, and alerting at least on those directly exposed to reduce/eliminate blast radius
2. Good engineers don't enable MTCP (which as far as I can tell is largely a desktop feature) on firewalls
3. ???
4. Profit!
Is step 3 reading the CVE for long enough to understand that disabling mptcp is an incomplete mitigation? What happens if someone discovers a bug in the particular firewall software you're running?
That was preventable the same way the big corps prevented. You have competent process in place to block at the edge and fan out rollout asap.
Combined with defense in depth across the perimeter. Come on yall, we've only had these tools at a high level for maybe a year and we have already forgotten the basics?
The same AI that can be used to find exploits in things can be used in an internal adversarial way to self-audit your code before you put it into production or expose it to the world. This will have to become the new normal.
Abliterated gguf of glm5.3 size and capability class models and similar are widely available for download, or the methods to produce them yourself from open weights safetensors files is also available.
I'm curious what people think of BYOC (bring your own cloud) as an option offering to customers or buying it as a customer. Looking at Northflanks website they list it as an offering, I presume those customers are not affected by the breach.
What are your opinion on the concept? Growing trend that should be encouraged or a mistake for most startups? Would you choose one provider over the other because of it assuming price was about the same?
I get that this is often a super expensive "call us" option today, but in theory it doesn't have to be, so for the sake of discussion let's assume it can be delivered at a reasonable cost for the provider at reasonable price for the customer.
If people from Northflanks is reading this, I'm sorry for the incident and I hope you recover as soon as possible.
Depends on the nature of the breach, if its a flaw in the software itself, then perhaps exploitable both in the hosted offering and BYOC offering, though perhaps the BYOC aspect gives a bit of "security-through-obscurity" and less discoverable as a target.
I was thinking of a fully private deployed within the customers cloud account connected to the software provider via Private Service Connect or PrivateLink and then either only available inside the customers private network or only via their own firewall. The point being that it's not publicly exposed.
We are increasingly seeing deployments trend in that direction. But trust goes both ways and putting systems in place to prevent abuse from either party is hard and needs to be thought through.
Unfortunately, this is going to be an everyday story. More and more bugs and breaches will be discovered and this move fast world more and more will be continued to be made.
Security is a perception, but right now it seems like it needs good PR.
How these Paas (Railway before) are being breached? Poor choices or something else? Scary Nonetheless. I thought AI advancement will make the self hosting painful. If PaaS are no good, then what can be done?
Modern PaaS is tough because there's a general expectation that users (and agents) are able to see and read back environment variables in their management UI or CLI - which means that you can't segregate credential values from the public web environment. See e.g. the visibility icons on https://northflank.com/docs/v1/application/secure/inject-sec...
And sure, you can encrypt at rest, and perhaps even enforce that only re-encrypted versions are ever seen by the management/API web server. But fundamentally this display requirement means that if your management/API web server is compromised, users' secrets can be obtained by a process of impersonation.
Stuff made during the move fast and break things era now doesn't hold up very well in the AI era. Or to say another way: startups are very vulnerable now - this new environment is unforgiving towards cutting corners and/or having inexperienced devs
I think the ultimate fallout will be that software "engineers" will finally have to become "engineers" in the real sense, i.e. professionally licensed and personally responsible for the designs they approve and the software they build, whether the actual code is produced by LLMs or humans.
As costs and broader economic impact and perhaps even injuries/deaths related to software breaches continue to grow, there will be growing intolerace for vulnerable software, just as there was intolerance for collapsing bridges and buildings.
And I'd also at some point expect something like a "building code" that applies to software that will be exposed to the public.
If anything, I see us moving even further away from professional licensing. The field's rate of change was the main reason behind a lack of any professional body, and that's only accelerating. I expect compliance audits like SOC2 to actually get teeth, though, and for organizations to be expected to actually fix issues instead of the usual security theater.
Why would ai advancement make self hosting painful? Apart from increased cost of hardware (which you implicitly would pay for in PaaS as well anyways).
If anything, AI makes running linux boxes much easier than 4 years ago; it's now faster and the learning cliff is gone to get the correct CLI commands and digging into system logs.
I had this exact thought yesterday. Shouldn't AI obliterate PaaS platforms like this one since it's super easy to build a bespoke CI/CD process now? Since it's bespoke, you can lock it down as much as you want. Attackers don't even know it's there. It's time to go back to `deploy.sh` and a VPS.
When you build something small, you can easily see all the doors and properly guard them.
As you build something bigger and bigger, quickly, with several people building, guarding all the doors becomes exceptionally difficult without a kind of aggressive discipline which is contrary to the "go fast" culture.
Cybersecurity in practice until now, for most companies, has been an insurance problem, not a quality problem.
Such companies are quickly discovering that insurance is a post-hoc partial correction and not a proper solution.
This situation will not improve anytime soon for most companies.
The solution does not require any new magical tools.
Run agents and everything they build. Secure execution from the first sandbox
Getting hacked while in the business of selling a secure sandbox isn't really a good look. It seems like can't even trust anything these days. The other day OpenCode got hacked and leaked everyone's secrets inside .env to attackers and now this. Sigh.
i wonder if there is a chain reaction point, when one compromised vendor holds secrets to another one and that becomes the next target. Northflank lists Sentry as their customer. So if Northflank secrets are decrypted and Sentry used them for deployment, you have a starting point to attack Sentry next which then opens up a lot more.
So if you want to judge as a customer for selection and figure out how likely a new(ish) PaaS company going to get breached, how would you evaluate them?
1. Ask them about what percentage of their own resources goes towards cybersecurity, audits etc. Would they even tell you.
2. Technical stack maturity. If the details are available, because they won't reveal any of it to you.
3. Other signals: Do they give talks on platform security, who is incharge of their cyber security.
There is an entire practice of cyber risk analysis, and most of the companies I work with do this type of due diligence as part of their vendor management process.
Yes:
* you do ask about things like budget allocation, audits, and internal compliance processes.
* you do ask about the tech stack, how it is maintained, and what the patch and disclosure policies and SLOs are
No:
*you do not rely on what figureheads say publicly; none of that is binding in the way legal agreements are
And if you're a vendor, your deals desk probably has all of these answers packaged up and ready to go; or even published on a website like this https://spyglass.trustshare.com/home (no idea who that is, it's a random company's compliance portal I found while googling my employers compliance portal provider).
Not in English majority speaking place at the moment so not surprised "runtime platform" for "autonomous software" is not immediately understood as the somewhat etymological contradiction that it is. Runtime doesn't even have a definition, really. but autonomous implies independent and not ran / hurried like bison off a cliff...
Unfortunately I'm not longer using PaaS tools. Instead I use Azure for work and vps now for all my personal stuff. With AI it's easier than ever to lock a server down tight and mitigate this kind of exposure.
I'm actually pretty pissed at this now. I have a throwaway appstore app that uses northflank for the backend, now I have to go in and rotate revenuecat secrets, apple secrets, ai secrets. For an app with basically 2 users.
Can I even delete the app on apple's app store or whats the process like what a pain in the ass. I'm pretty done tbh.
I feel like these breaches started to accelerate after Anthropic wrote a full post about how easy it is to abliterate glm 5.3 to make a frontier-level cyber model.
Now, to be clear, abliteration was no state secret, but that blog post was pure accelerationism.
That was just 30 days ago. This has been happening for months. Part of the "Black October" narrative is just vulnerability research; Calif didn't abliterate 5.3 to find PHP2Shell.
Yeah IIRC calif research on php2shell preceded all of that but I still stand by the thought that Anthropic has been the most accelerationist AI company despite its lip service to the notion of safety.
I think they definitely boosted usage. I certainly started using the -cyber models defensively against my own systems. But defenders like me are usually late to the party. I think the attackers have been here for a while but that the technology reached performance only recently.
The idea that attackers were waiting for Anthropic to tip them off on this is kind of risible. It's been a huge topic amongst practitioners for the better part of a year, and practitioners are usually a step or two behind dedicated criminal or state-sponsored attackers.
Right, that's kind of my point. Also: it's a good blog post (the Anthropic one), but it observes abliterated 5.3 models were already public. It's not like people who break into computers didn't know that already. They're not waiting for Anthropic to tell them about open weight models; they already live in the world of open weight models.
Duh, the idea that those capable of actually utilizing these models for these purposes weren't aware that they could use these models for these purposes up until Anthropic released a blog post about it is silly. Anybody who is even a bit tech literate has been well aware of this inevitability for basically as long as open-weight models have been a thing.
The timing of all of this is much easier to pin to the creation and proliferation of capable of open weight models. Anthropic calling this out is not the cause for it; it's just an obvious correlation.
You are running a machine with tons of vendor provided agents/scripts to do god knows what on a cloud provider which itself runs tons of agents/scripts to manage that infrastructure and then you expect to be secure.
And no, before the labs there was a degree of fear for everyone to hack into some one else system. With the labs changing the narrative, anyone can just blame the agents for the breach, whoever did it would only get fame and no blame
I hope this is a real wake-up call for teams running anything remotely SaaS-shaped. I count at least six substantial, lightning-speed attacks against multi-tenant as-a-service vendors since September. Not this year--just since last month. PingCAP/TiDB, Fireworks AI, Zeabur, OpenCode Console, Railway, now Northflank. All of them modern "we keep your content safe" players with high hopes and probably pretty good protections.
This is a full infosec emergency unfolding here. No matter how grinding it is to motivate a team to invest in protection or resilience (and it is grinding): We gotta get ready. Stat! The AI-accelerated attacker apocalypse has begun. It's boom, boom, boom, thanks for everything, on to the next. They have us sighted and are firing for effect.
So credential rotation, tripwire secrets, DFIR on standby, solid vulnerability and intrusion detection, real listening to that signal, minimized blast radius, spending caps and time limits and tight scoping on credentials, fast incident alerting... We need to get on a real war footing here.
War ?
Last I checked major companies are still outsourcing essential engineering to save a buck.
A bit old, but how was this practice ever acceptable.
https://www.propublica.org/article/defense-department-pentag...
A multi trillion dollar company( Microsoft) too cheap to hire onshore for DoD contracts.
Worked at a Fortune 200 company. Any govt project you worked on, you had have to have a basic civi security clearance. Any govt contract we got specifically stated you couldn't have off shore folks working on any of the code. Not sure how MS got around those requirements.
You gotta get to the Fortune 10 before the real leverage kicks in! On the other hand the fortune 200 company I worked for briefly had many signs of deep seated corruption. I couldn't stick around for a multitude of reasons, but the "follow the rules wink wink" attitude was flabbergasting, I'm sour on publicly traded companies ever since, how do you know you're not investing in Space Balls?
Fuck, glad somebody else is seeing this.
burningChrome is broadly correct, but the devil is in the details.
There's quite a history of US federal programs encouraging critical federal systems to operate within private infrastructure. This once required very strict scrutiny, joint DoD/DHS/GSA reviews, and was generally a long/expensive process that created the perception that a few providers had a gotten a hold over various agencies through these outsourced operations, which, combined with a fresh "interest" in federal efficiency, influenced a recent national defense budget authorization to formalize a slew of processes with a few minor tweaks as the law of the land, with a tweak or two to allow new cloud operators and service providers to authorize quickly and operate efficiently, particularly for low-risk classes.
Today, this means SASS providers must find an engineer with clearance (not just authorization to work in the USA). This cleared programmer documents and authors a whole host of (typically new) procedures, such as authenticating against a "federal enclave" authorization directory, delivering changes authored upstream to a government-controlled CI/CD system, and working with the SASS provider and the Government Services Agency (GSA) to ensure the backend operates when running within something that could be compared to a special AWS AZ. Things change upstream, and our operator prepares a 'significant change' request, which is, if important, gated by a liaison from the GSA. No one is responsible for scrutinizing implementation details of these 'significant change' requests, nor are their authors required to provide any information about themselves or the code's provenance. Given that hundreds of programmers may have written code included in said change (which is typically already deployed upstream), this is often efficient and generally fine.
Generally.
Easy. They hired Americans to be actual keyboard people but they didn’t know anything so they were basically a KVM. However Microsoft could say “Yes only Americans were making changes. “
I guess like that: “we are Microsoft, 95% of your things runs on us, you want to stick to the rules? - OK that’s additional 10% on each and every license you have with us. You are still here or are we done?”
Step 1. Find code on GitHub.
Step 2. Copy it into your repo. Who cares what the license says, how would the authors of that code ever see it being misused, because the place it is being misused requires a security clearance?
Step 3. Deliver the code as yours.
Truly, how will the people who authored something on GitHub ever know? Any circumstance you describe would require someone without clearance to know implementation details of some software. Which is to say, either the thing I describe happens all the time, or security clearances are overblown.
It's been covered before.
Digital Escorts
[dead]
I wonder what it's like inside the big cloud providers. I think outside Microsoft they've always been security first, but this new era has to be crazy town.
Google at least has been investing very heavily into cloud security. They acquired Mandiant ($5.4 billion) and Wiz ($32 billion). Many people felt those were very expensive acquisitions at the time, but maybe it seems like a wise move now.
Those are standalone products and professional services, not internals they use
A problem is some people and teams are world class at a big company... But not everyone, everywhere. For a fun time, read about the rollout and internals of Azure.
This sounds really interesting — have any links you wouldn't mind sharing?
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
I mean, Microsoft and Security, those two words have never really been associated with each other in a serious conversation.
They're quite famous for being lax in security - mostly because they're a massive corporation, but also because they're high profile
That's why the comment said "outside Microsoft".
But keep in mind Google got hacked by the Chinese in 2009 or so.
Apologies - I did misread
“besides Microsoft”
Sure. But that's not the words they used.
Back when APTs were getting numbers, the joke was that Microsoft was "APT -1". The presence of windows/excel/outlook in your network was more of a threat than Anonymous, China and the DPRK put together.
> All of them modern "we keep your content safe" players
But they weren’t and probably aren’t and won’t be. They just put forth an image as much as needed to sell.
It would seem Google account details have been compromised or otherwise correlated, with not a peep from Google, we are getting reports of scam calls to people's real phone numbers about the "business opportunity" available in their attached Youtube properties.
Another possibility is advertising shadow profiles are able to correlate logged in accounts across YouTube and other properties.
Is this the point where most tech people accept that anything a closed source Software company says about security is effectively a "Trust me bro" as long as it is not backed up by independent verification?
We love to talk rubbish about finance bros but that industry has had independent auditing as part of its DNA for a century[1]
[1]: https://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act
I am unclear why you linked to a 20yo law that was enacted a few years before we found that the entire sector was floating on loans to people with fake information.
Should startups that have experienced a data breach face a lower valuation in subsequent funding rounds?
I wonder which one of these are at least partially due to effortless vibecoding...
Also, if vulnerabilities are so easy to find, it should also be easy to detect them through a simple red team exercise.
> probably pretty good protections
In my career I've seen maybe two companies have pretty good protections. Both had government contracts so they had no choice. Otherwise there's no reason for companies to secure their stack. They make money easier without it.
The buildings are on fire because they're being built out of straw because it's cheaper. Only a government regulation to build out of brick is going to stop the fires.
I for one have backed up to offline storage our research lab's data in preparation...
A war footing?
Yes. Literally AI is being used by actual nation state adversaries to compromise bits of infrastructure that companies rely on to run their services. They are compromising this infrastructure specifically because they seek to gain advantage against their adversary (us).
Sorry... our high-T Spartan chad warfighters don't do computers, that's nerd shit.
> ...the operational and economic condition in which a nation, military force, or organization mobilizes its resources, industries, and policies as if an actual state of war exists...
Yes?
> … The Congress shall have Power . . . To declare War, grant Letters of Marque and Reprisal, and make Rules concerning Captures on Land and Water;
Can’t say I’ve seen this?
> as if an actual state of war exists
"as if"
So, because congress didn't declare it, the war doesn't exist? Lmao
> SaaS-shaped
The usage of shape here triggers me.
Your words are comment-shaped.
Maybe Bill Gates has a point when he says that models capable of significant cyberattacks need to run on controlled infrastructure. Is the era of open weight models you can strip safeguards from turning out to be so great? From my perspective, this kind of sucks.
Sorry but how in the world would that help with anything? You can't control an adversary, that's not how adversaries work. So that makes what you said a literal cop out.
Are these cyberattackers using models they developed themselves? Or are they using models they stripped safeguards from? It’s the latter, right? And it’s only going to get worse if we don’t take steps to control the technology.
No. The Chinese have already developed frontier models. And we would be fools to assume that's a permanent end to the list. North Korea has nuclear weapons. Iran has ballistic missiles and leading edge military drones. There is no fundamental reason why both countries shouldn't be able to train frontier AI models.
Trying to protect ourselves by putting restrictions on the technology would be like trying to hide from a tiger by wrapping a towel around our heads on the basis that if we can't see it then it can't see us. That trick only works on imaginary monsters, not real ones.
No, that cat is out of the bag. First, unilaterally forcing controls on models will not shut down open weights produced in other nations. Second, the capabilities are too powerful for military amd spy agencies to ignore. They will force the creation of unshackled models to attack IT systems, even if it must happen in secrecy.
You should assume any of those are possible. And by that I mean: it's not physically possible to "control the technology" without instituting total surveillance and even that's just false security.
Defense is the only way.
You don't realize it, I imagine, but you're essentially advocating for the contemporary equivalent of requiring a license to print text on paper and distribute it.
We are not helpless. We can choose to control the technology and do so in a way that is in line with democratic principles. After Hiroshima, people assumed that most nations would develop nuclear weapons and that nuclear war would be commonplace. But Nagasaki was the last nuclear attack, and 80 years on, only a dozen nations have nukes. This technology makes nuclear weapons look like toys. We either choose to control it or we will be the victims of it.
I feel like this is quite a different situation. Building a nuclear weapon requires a lot of things to happen in the physical world that we can watch for. Weapons grade fissile materials must be sourced or refined, facilities must be build, you can't just do this in any machine shop. Various experts must be either poached or subverted or home grown. There are a lot of people with mouths involved. Then once you get past all of that you either have to test or just hope for the best at which point you will certainly be detected and likely handled with extreme prejudice.
Digital intelligence seems much less controllable. Near frontier weights are already out there, you don't necessarily need massive datacenters if you're not trying to serve commercially. Even if we we're somehow able to control what OTHER nations do through either treaties or threats of sanctions/force, working on this stuff is easily in range of many of the larger criminal orgs. Beyond all of that, there's always the chance that weights from authorized providers find their way "out the back door".
We've seen with crypto and darknet markets that even if you can fight this kind of stuff and catch bad actors, the most actual change you're going to effect is a temporary slowdown and the next guy who tries is going to be that much better. I don't doubt that regulation is going to be the initial route this goes, mostly due to the financial interests of the frontier companies and the desire to keep this build-out solvent for as long as possible rather than any true concern for actual safety and sanity. What's really going to need to happen though, in my very insignificant opinion of course, is we are going to have to change from the ground up how our digital infrastructure is built.
Maybe with the coming changes that is for the best too, our current systems are designed for people and I don't know the current numbers but it seems like agent traffic is already the bigger percentage. We may not be able to build our infrastructure on top of these large mountains of human designed legacy code. Maybe we need to be thinking about new kernels, new servers, new protocols and stacks.
We seem to be very quickly reaching a point where is something is not mathematically provable to be secure then it is not secure at all.
This is much more akin to books than it is to nuclear weapons. In my view you're proposing to tightly control access to something akin to the next generation of books. Is knowledge power? Yes it is and it's much more dangerous concentrated than distributed.
Buddy you're nearly 20 years too late for Bill Gates to be making any decisions at MS.
Unfortunately for infra startups, I think we're going to see a convergence back to hyperscalers as a result of incidents like this and railway.
There's no amount of "nicer to use" that can replace security, at least certainly when selling to enterprises
I actually think it’s the opposite. I think people will move back to their own hardware with OSS deployment like dokku. If everything is more bespoke the attack surface is smaller as is the pay off. Hacking railway got access to lots of api keys, hacking some random vps gets less payoff.
> If everything is more bespoke the attack surface is smaller
This is exactly the problem that LLM assisted attacks solve. For a human attacker, the extra work to figure out a bespoke solution is too time consuming to be worth the effort. LLMs can search for vulnerabilities and spit out bespoke attacks like it's nothing. On top of this the bespoke solutions weakness leans heavily into the AI attackers advantage: response time. Even if they can just get your company's single Anthropic key, if they do it at 2am while your one infra person is on vacation, they have plenty of time to extract enough value for their effort.
I agree with the move to hyperscalers. The future (well, actually present) is going to be constantly seeing new attacks happening. The only real solution to this going to be having teams of people and agents constantly watching for these and able to deploy fixes and respond to risks immediately.
You’re technically right, which is the best type of right, but my point is the fragmentation and lower reward will make discovery harder, and the value of searching and cracking less worth the effort.
I had a html game I built for my niece on railway with an open ai API key so she could vibe code changes. No hacker would have looked for it but now I am catching strays because railway was a juicy target. Hyper scalers would be the same. Some random colo not so much.
>and the value of searching and cracking less worth the effort.
This is depending on the cost of tokens not trending towards zero. Maybe not today, but in a couple of years I'd expect hacking models on asics to reduce this to almost nothing and the systems being very competent in using deterministic tools very well.
> LLM assisted attacks solve
Yes, LLMs can easily knock over your on-prem docker appliances.
But you can still put stuff out of reach. I'm currently designing a key on-prem appliance that is can be compiled down to transistors, giving deterministic wire-speed performance while ruling out RCEs. It's cryptographically homomorphic: it processes authorized inputs but doesn't know what they mean, so an attacker who obtains the disk doesn't benefit either. The design will be highly resistant to exploits, and is likely to remain outside the "next-token" prediction training set.
This is the kind of thing AWS won't sit down and engineer for you. But thanks to LLMs I finally have the time.
How is that ever going to work out? The speed of discovery for LLMs is so fast, these places are getting breached more often than not due to a human errors. Its really hard for machines, let alone humans, to guard against a lot of LLMs going against them 24x7.
It's much easier to protect yourself if you have your infra behind a perimeter firewall you control.
Until there's a exploit for Palo Alto, Juniper, Cisco, F5, <pick your defender>.
You can scratch Cisco from the list - they just announced they will re-vibecode all of their codebase! So there will not be any explots for their hardware in the future - one way or the other. ;-)
This solving the problem once and for all.
My hate for hyperscalers is greater than my hate of being SRE.
It's time to leave GCP, AWS, and the rest.
Google and Amazon markup can eat it. Fuck those guys.
Let's not forget that Google and Amazon tried to offshore us, played a role in inventing the labor-displacing models in the first place, and that don't give a shit about any of us.
I hope they ossify and turn into the next IBMs.
They thought AI would lead to growth and a greater concentration of wealth for themselves, but it's actually the seeds of their own destruction and dismantlement.
Or there's some exploit in an endpoint device, and the problem moves laterally from inside your firewall.
Yep, but the attack surface is much smaller!
yeah remove the door locks, the thief can just break the window
It's also harder to explain to your customers why you're the only product they use that has a security problem when an exploit does eventually happen.
You see this already with uptime. If Github or AWS has a major outage, it's in the news and even people that don't understand software realize it's just "something happening today, everything is down".
And, unfortunately, in practice customers don't care about this inverse, when "everyone else is down but not that one product".
Good like trying to explain to your customers that their data is no more accessible because some api key ended up in a public git repo or since you admin panel is exposed to whole internet someone was able to log in.
With the speed of exploitation hitting right now your analysis will be out of date in a few weeks to months. We are about to have very palpable fear hitting all software vendors. It will be seen as irresponsible to use centralized systems
Especially when the customers' own websites or services are down, they are a lot more understanding.
Until the attacker bypasses your perimeter firewall by convincing an employee to download and install malware.
And so? This is true for any kind of infra!
You can't defend yourself from everything, but you can try to make you life easier by reducing the attack surface and make side movements more difficult.
Having employees (or system like bastion hosts) with unrestricted access is one of the recurring mistakes in such events. I can’t understand it.
Zero Trust is expensive to implement and the business people hear "at the end, the systems will ideally work as good as before".
It will take a few bankruptcies until security is taken seriously.
[dead]
No self-service sign up for your app. No public access to the app itself. Website that let's you submit a sign up request form. Team reaches out and reviews your request, creates an account in the app dash for you, and you set up mTLS to get access to the dash.
In-person requests for access to any app. Request interview includes word association exercise that monitors your emotional reaction to a string of key words.
Security question 2: You reach down and flip the tortoise onto its back…
You joke, but the space I work in actually does do video calls (or in our case in-person meetings if they're local to SF, often) before onboarding new customers.
Having them hold 3 fingers up in front of their face?
LLM discovers exploit -- you don't realize it in time because you don't have a security team as big as the PaaS you used to use -- you get pwned anyway.
This has happened to many high-profile self-hosted pieces of software just this past month.
An exploit in what, the network stack of the Linux distro you're using for the mTLS proxy? Or the crypto module? Sure, could happen. But then, you're unlikely to be the top target of anything.
Or do you mean a customer was running it?
There have been a lot of exploits for a lot of software on-prem tech stacks rely on just over the past month - proxmox, php, nodejs, palo alto, eMudhra, etc come to mind.
> But then, you're unlikely to be the top target of anything.
That would have protected you 3 years ago when a human had to bother to attack you. Today it doesn't protect you because it's trivial to deploy an LLM to act like an APT against any random target that shows up in shodan.
It takes more effort to secure on-prem software today than it did just a couple of years ago. Updates are more frequent, zero days are more frequent, attacks are more common, more persistent, and more automated.
I think most people responsible for securing production systems today will share the experience that their workload has increased.
You're still describing an environment that presents a pre-auth surface area beyond a listening port.
And I am literally a security engineer; the work has absolutely not gotten harder, it's just changing in how you think about security, just like it shifted before from "secure the perimeter" to "defense-in-depth", it's now changing again to focus on automated response and compartmentalization/ segmentation (and I mean real segmentation, not the checklist-style "we have 3 vlans, but they can all talk to each other" 'segmentation' you used to see.
Yeah, most organizations need to expose something to the internet as a matter of normal operations -- at least some sort of a web app if not dozens. It would be great if we could all just have udp 56820 listening silently for a key, but that isn't reasonable for most organizations that have normal users expecting normal applications.
Yes, everyone is rushing to implement zero-trust principles for real now. It's getting harder to trust that your public facing endpoints don't have vulnerabilities, and it's getting harder to trust that your internal software and endpoints haven't been compromised from within your pre-existing trust boundaries. A lot of the increased workload is due to patching up the existing security infrastructure, and on top of that, building out new solutions to better fit today's threat landscape.
Ideally they need to divide those apps entirely; your public website doesn't need to talk to your app.
For stuff like Netflix that is truly consumer-facing, where mTLS or IP whitelisting are non-starters, the reality is you have to rethink your model entirely.
Also, I'm not sure where you got the idea I'm talking about on-prem only; this also applies to using IaaS/ cloud hosted infra.
The reason we were talking about on-prem is because we're talking about the effort of doing these things yourself vs farming it out to a third party.
If you're a security engineer responsible for securing only:
1. a public website
2. stuff that can be ip whitelisted and/or use mTLS
Consider yourself very lucky. Most orgs need to expose a lot more than that.
Almost no companies farm their security out to third parties. The era of MSSPs is long dead, except for small non-IT businesses like mom-and-pop law firms.
> Most orgs need to expose a lot more than that.
No, in fact there are only 2 kinds of connections:
1) public, necessarily allows anonymous/unauthenticated connections
2) non-public, doesn't need to allow unauthenticated connections
Name a service, and I can tell you which one it is.
Or put another way, there are authenticated and unauthenticated services. If you can authenticate someone at the application level, you could do it at the network level.
Most places have combined them at some boundary, for ease of access, but there is no case in which an authenticated application endpoint (even /login) has to allow unauthenticated network access to it.
And in all seriousness, name a service that is not a public website, but cannot use mTLS or IP allowlisting.
SFDC? mTLS. Customer webhooks? IP whitelisting. Data warehouse? PtP VPN.
> Almost no companies farm their security out to third parties.
They do it all the time, piecemeal, that's part of what anything-as-a-service delivers. If you don't want to secure an app, buy a SaaS one. Don't want to secure infrastructure?, buy an IaaS.
> Most places have combined them at some boundary, for ease of access, but there is no case in which an authenticated application endpoint (even /login) has to allow unauthenticated network access to it.
> And in all seriousness, name a service that is not a public website, but cannot use mTLS or IP allowlisting.
Literally any SaaS software where the customer does not or cannot practically install a certificate for you or provide a list of IPs in advance? mTLS is almost never used in a browser unless you are connecting to a government website with a pre-shared smartcard. IP whitelisting is great if the end users only work from behind a VPN or in an office. But there's a whole big world of people who cannot or will not do that.
And that is going to have to change.
There are already plenty of methods that are low-impact to users, like certificate-pinning in apps, and all you have to do to make it per-user is roll in their user-specific key JIT when they download the app.
Users will not be happy (just as no one is happy Reddit is going app-only), but they will do it, especially when it becomes common as companies decide they don't want to be the 5th breach notification on HN in a given day.
Today we have Bun, Cursor, Northflank, and now Linode!
Or the configuration parameter that you got wrong because you rolled your own and didn't know any better.
Why would you roll your own?
You would inherently be rolling your own configuration when deploying something on-prem.
1) where did you get "on-prem" from?
2) that's not what "rolling your own" means. Building your own web proxy is "rolling your own". Standing up e.g. Caddy and configuring it is not.
1. that's what this thread is about?:
> I think people will move back to their own hardware
> self-hosted pieces of software
2. arguably -- but lets set aside what you think the word means and respond to the obvious meaning of the comment you're replying to.
The point is -- doing shit yourself makes it your problem to do correctly and requires sufficient resource allocation
1. That's fair, I didn't realize people were expecting my comment to be perfectly-aligned to every aspect of the thread parent, including the "their own hardware" part, but now I at least see where the confusion came from. For the record, any issues at this level of the stack are also present in IaaS, not just on-prem.
2. I still don't see where you got that they are only talking about configurations versus messing up an implementation (I.e. in a paramater they coded incorrectly)...
But that aside, even on AWS you're still configuring your proxies, gateways, load balancers, etc. AWS gives you just as much rope to hang yourself with as a box in a colo network cage does. Companies are already doing this stuff themselves, on-prem is not more security work. IaaS is about scaling and maintenance load, not security being handled for you.
For one, I have strict iptable rules to drop inbound from outside my specific commercial subnet and my specific residential IP. After that, all the usual best practices.
Breaking a lock still costs approximately the same, whether the lock is protecting a small barn or a huge hangar. A large number of small but reasonably tough (not unbreakable) targets makes the payoff per target smaller. This necessitates target selection, and chances of being deliberately skipped grow for an average target.
Everyone knows Fort Knox has gold, and where it is, that’s railway, heroku. No one knows where my personal safe is or whether it has anything worth taking. That’s a vps in a colo
However, the large hangar is likely to have armed guards inside, or at least able to arrive on the scene within two minutes of the alarm tripping. Your regular household with their regular locks on their regular doors does not.
This is absolutely happening. I'm not a developer in SV, I'm an IT leader out on customer land, and I can tell you a LOT of companies are looking to bring things back on prem for security and cost reasons. And before anyone laughs, yes, you ABSOLUTELY can save money going on prem, I've done it for company after company. It's one reason the PE firm that I've worked under for a decade kept moving me to new companies.
Let them laugh. Self hosting is much easier with OSS that solves most of the pain that made the cloud transition worth it, and commodity hardware has been cheap and capable for over a decade.
Sadly these relatively recent price bumps on hardware have made on-prem a bit of a harder sell for some.
My company just did this. Still using AWS S3 but everything else is on-prem with kubernetes.
That sounds insanely expensive. AWS S3 only makes sense if your workload is also in AWS. Using AWS S3 while having your compute elsewhere is a recipe for an insane egress bill.
Dependa how you use it. If it's only for backups then it's fine.
How much of your public facing workloads rely on S3 if I can ask?
There's plenty of OSS drop-in replacements for S3.
This is misguided thinking. You are less likely to hear about on-prem breaches in the news just because they are less likely to be newsworthy incidents, but they are even more likely to happen. Part of the reason people moved to the cloud is because it is much easier to have a team manage updates and security at scale.
Being a small target is not helpful these days, attacks on on-prem software today simply scans the internet, or uses databases by people who have already scanned it, and is automated.
On-prem software has not been spared from the recent spat of security incidents.
There is a difference of the security of software running in your cloud resources rented from AWS etc. managed by you and being a tenant in many SaaS systems all cobbled together all running on rented hardware. There is a lot more attack surface and a lot more diffusion of responsibility.
My comment[0] tries to explain the nuance and I would agree with you and both the GP in some sense.
For Entreprise, the shift will be towards hyperscalers as they want trust and less blame (Nobody got fired for picking AWS)
For consumers, we care about pricing and stability and control and we will move back to our own hardware or cheap VPS providers and self-host things.
The middle ground would be where the slight pain could be at in my opinion.
[0]: https://news.ycombinator.com/item?id=50307410
> For consumers <snip> self-host things.
That's not how consumers work. That's a nerd/geek/hobbyist mentality.
If Ikea and Wayfair went out of business you wouldn't expect consumers too take up woodworking and build their own furniture en masse.
For consumers/small orgs, how is the security at a cheap VPS provider going to be any better than at a cheap IaaS provider?
as long as you follow the herd (for example, disable password ssh login, only allow ssh key login, use something mind numbingly boring like debian stable, set up unattended updates, update and reboot every night, etc, don't use any exotic packages etc) I suspect you are simply not a worthwhile target even with LLM agents as opposed to humans
Until your VPS provider has its management dashboard breached, allowing an attacker remote console and configuration access to your VMs and everything that enables.
I dunno why you're getting downvoted, they don't even need their control plane breached, it could be a neighbouring VM running something out of date and an attacker getting out to the hypervisor and taking your disk images without you or the hosting company ever knowing.
Then buy hardware and go colo. a dell server off eBay would far exceed what most smbs need.
It's probably both.
Cost of hacking a custom setup is going down rapidly. There was broad agreement that the PHP vulnerability was very difficult to exploit in practice. It took two hours for someone to get into Steal My Bitcoin. Apparently all they did was tell a decent model "go exploit that." AI isn't free but it's not that expensive.
If you have to hack 1,000,000 custom setups to get the same results as one hyperscaler, would you bother?
Depends on what takes more effort. It's not at all clear to me that the 1,000,000 custom setups takes more effort. The first few will take some, then you'll start to see patterns and it'll get easier as you go. Custom setups aren't going to be all that unique in practice.
No thats an old way of thinking because it assumes mass targets are better because hacking is a high effort activity. But with the automation of intellect the unpatched systems (which a much more numerous when everyone self hosts) is the key point.
Mass credential stealing is just the top funnel of hacking and (compatativly) isnt that lucrative - these credentials are then sold on the black markets to the next step of hackers who use them to exploit targets.
With unpatched systems and self hosting, this second group can just set an llm at a target and have a much higher chance of success without needing credentials
It would be a dream for the infiltration hackers to have everyone self host. Unpatched and miscondigured systems would be everywhere and LLM automation could make mass scanning them and exploiting them very attractive - not mass scanning and exploiting them for credentials stealing, no! For layer 2 stuff
> There's no amount of "nicer to use" that can replace security, at least certainly when selling to enterprises
Azure would beg to differ.
https://news.ycombinator.com/item?id=50301854
None of my customers / clients / coworkers have ever used Azure and "nicer to use" in the same sentence.
and it wasn't what he was saying, the "nicer to use" in the context were startup IaaS alternatives
A moron could build a better UX and DevX.
I have a bridge to sell you if you think everyone on the Azure team is sub-moron. Infra security is a hard problem to solve, GCP and AWS don't quite have a perfect track record either, perhaps AWS stands out as a better option but it's hard to argue about it.
I don't think the blame lies with the intelligence of the Azure team. I would argue that there exists structural flaws in the business incentives which deprioritize 'users' from 'customers', along with the stark lack of consequences for security failures.
That's a problem across all of tech, I know a startup that recently had a security incident, lost over 40K USD and none of the heads have rolled they haven't completely revamped their security posture or anything like that.
While there are incentive problems across all levels, The political quagmires that make it impossible to move at any reasonable speed, and that lead to counterproductive ideas getting large budgets are in my experience very common at the highest sizes. It's true that on the other edge of the spectrum, small startups make poor choices, but many of said startups will fail due to said choices. So by the time you have a viable company making good money, the really badly aligned companies are gone.
So when I look throughout my long employment career for best security and organizational practices, all the highlights are still relatively, relatively undersized companies, with headcounts in the 3-4 figures. It's as they grow, and start to be multi-headed behemoths, that they decay again, as what is good for the responsable mid-level exec, for the business unit, and for the company as a whole start to have nothing to do with each other. That's when you can smell the decay in the hallways. A few large companies manage to stave off this rot a little more than others, but it's inevitable. Then nothing important can be done in a timely fashion.
> but many of said startups will fail due to said choices. So by the time you have a viable company making good money, the really badly aligned companies are gone.
I wish that was the case, but that isn't how startups fail generally, sure very bad user facing breaches could cause something like that, but lots of platforms with too much jank have somehow survived all of these cyber vulnerabilities before, and will probably keep at it.
For a majority startups it's about connections and relationships rather than competence. Very few even consider building actually useful products to be a priority after the initial work of building it.
40k is not a lot. Honestly that sounds like something that should have no consequences.
Places where it’s less of a problem are places I’m spending my money.
and anyone else’s money I’m asked to spend…
It’s about incentives. My company spends about 400k/month USD in both Azure and AWS (different workloads). When I ask my AWS team for help, I usually get actionable info, a separate standalone call (if the request warrants it), and a follow up from my TAM and Account Exec. Every request to my Azure team has basically been “here are some YouTube/MS Knowledge links” + let me know if you figure it out. Roughly the same spend on both platforms, and wildly different levels of helpfulness (and yes, we have an Azure EA).
Are the people on my Azure team smart? Probably. Are they incentivized to actually help me solve a problem? No. They’re obviously incentivized to increased my license spend.
Unless you're paying millions a month, they don't care. It can take weeks to get AWS SES (Simple Email Service) "approved for production" on a small account, and that's after dozens of emails to prove you're not a spammer. For an enterprise I was working with, I asked and 5 minutes later it was approved. No questions.
I'm sure there are absolutely brilliant people working at Azure. I think the overall system is very badly architected and I don't know what it would take to salvage it. Infra security is a lot easier to do when you have a system that has mostly solved it already, and I'm not saying GCP and AWS are perfect, but "security is mostly solved" is a decent description of their systems, I think. Of course it could all fall over tomorrow, but I'd guess that's a lot more true of Azure.
Security may be solved, but IAM is still a nightmare on both GCP and AWS. Nobody understands what all those privileges / actions / conditions / etc. do. Maybe LLMs pretend to.
AWS and GCP both work pretty well if you avoid conditions, which are a nightmare. Azure is like everything is conditions, and also buggy.
They work, but the documentation is overwhelming. With my last GCP project, I ran into security theater folks that didn't like predefined roles and demanded custom roles with dozens and dozens of granular permissions. Do you think they audited all those permissions? Nope. The simpler predefined roles were just not permitted by their checklists.
They implemented NVRAM in the UEFI and then didn't decipher what the NV means in it so it is not persisted. Their stupid virtualization can't handle not 1MB-aligned disk image.
Plenty of morons there
It didn't help that azure (especially in the beginning) was a cobbled-together mess of ancient Microsoft tech stacks that were never intended to talk to each other.
Are hyperscalers actually more secure though? I have only heard anecdotal thinking eg nobody ever got fired for choosing IBM.
It’s the same thing: let’s say Microsoft or AWS got hacked. Then you’re definitely not the only one who got hacked.
So, you can always say: Look, it’s not us (especially not my org!). It f’ing Microsoft/AWS! Everyone got hacked.
You and your career are gonna be just fine.
In a prior role, we realized something similar with outages. For our particular product, we AWS went down - we went down. We freaked out the first time, but after it was done we realized not a single customer had inquired. This felt unusual since we were getting emails quickly if we have a self-inflicted outage.
Turns out, when a major provider went down all of our users were too busy fighting their own fires. When they do notice, it's on a day when ALL of their vendors are struggling so we don't get singled out as a bad apple.
The whole "No one gets fired for using IBM" analogy.
This is especially true if the majority of your customers are also on Microsoft/AWS/etc.
Literally 0 of your customers will notice that you are down if their own systems are also not working.
This is perfectly valid for many businesses, up to a (shockingly high) level.
LOL "if my competitors are all down too, then it's OK." I guess there's a bit of sense to that.
I didn’t say competitors; I said customers.
If none of your customers are up to call your service, it does not matter if your service is up or down.
Yeah, if Exchange goes down, you get a day off. If the little boutique email provider you chose goes offline, you get a day in calls with everyone yelling at you.
It's today's "nobody ever got fired for buying IBM". You're really just want the the SLA (someone to blame when things inevitably fail)
If you need deniability, sure.
But it you actually NEED security, it's clearly not an option.
Azure (and AWS) offer Gov clouds, and probably on prem air-gapped deployment of their full stack of Fabric for that type of NEED security customers who can pay for it.
It is trade-off between risk and access/productivity with a cost dimension always.
Depending on your threat model you could go to typewriters /paper as some Russian department did famously few years back too.
I setup parts of govcloud.
The real sensitive customers are on oxide.
AWS (and others) have ladders of this was my point. GovCloud is just the one end of the this AWS also regions for Secret and Top Secret work which are reported to be air-gapped. There are probably programs beyond that where the customers are operating the stack fully on their own and so forth.
Indeed.
If you want deniability no one should be doing business with you.
You are just describing how they're a more valuable target for an attack.
With a lot of PaaS type vendors you are effectively exposed to both the vendor themselves and the public cloud they run their stack on top of.
Broadly I think you're much more likely to be the victim of a platform-level breach involving an immature startup PaaS than a hyperscaler cloud, however I think for small teams the alternate risk you take on is that AWS/GCP/Azure are full of landmines and extremely easy to misconfigure in insecure ways if you don't know what you're doing. Especially if you have AI telling you confidently wrong things.
When I was at Microsoft we took security cripplingly seriously to the point we could not deliver features and our competitors could because the security process was convoluted and had multiple reviews and approval steps.
Having crippling bureaucracy does not correlate with good security outcomes, except maybe negatively. However, having a history of unending egregious security failures, like Microsoft, does correlate with poor security outcomes. For that matter, having a present of unending egregious security failures also correlates with poor security outcomes.
Let us look on the front page and see if there are any, oh look here is one: https://www.binarysecurity.no/posts/2026/10/one-root-case
Having crippling bureaucracy does have better correlation with security outcomes than the “move fast, break things” philosophy that most of the software industry lives by.
I’m not even blaming Zuck for that, he didn’t really invent it, though he certainly popularized it — he just made an astute observation of the dynamics in software and turned it into a company motto.
Neither is correlated with good security outcomes against the prevailing threat landscape of profit-motivated, commercial, professional attackers with tiny budgets of a few million dollars. The standard is not "better", it is "adequate".
Correlation is not an absolute measure here, it’s relative. As is the entire concept of security.
As is what constitutes the standard of “adequate.” Which BTW is the standard in ALL engineering (not just security or software) because we live in a world where economic pressures apply and costs must be managed. Trivial example: bridges only use an “adequate” amount of steel and concrete because they are not free. The word “adequate” fundamentally involves the concept of tolerances. There’s a reason we consider over-engineering a bad thing.
So which development approach would you say is better correlated with good security outcomes?
Adequate is making it uneconomical, or cost ineffective, for threats to your organization to attack you either deliberately or indiscriminately. Absent a specific organization, we can use a profile of commonplace, prevailing threats as a proxy. A profile of commonplace threats is, as I said before, a profit-motivated, commercial, professional group seeking a multi-million dollar payout and thus being profitable at a cost of a few million dollars per commercial target. That is "adequate" in the prevailing threat landscape if you are the "average" organization.
That is a great definition of adequate for security! My point is that at scale, bureaucracy is a necessary (though not sufficient!) condition for achieving adequate security.
Another way to look at it is, bureaucracy is an aspect of the corporate culture, and lasting security comes down to having the right culture. At least in that respect Microsoft is doing something right. Doubtlessly a lot of that bureaucracy is ineffective, but it's a signal that the cultural intention exists.
As a relevant counterexample, we've never heard of these kind of processes at Meta and so you get Cambridge Analytica.
My perspective is skewed by my experience being on a security-adjacent platform team for years. Our biggest challenge was not external threats, it was internal teams seeing us as friction on their productivity, to the extent that it regularly led to escalations.
So when Cambridge Analytica happened, I played it up for all it's worth; Meta may move fast because they didn't have our "bureuacratic team", but look where it got them! Unfortunately the $25B penalty happened just last week... I'd have gotten infinite political mileage out of that.
the only reason you’re seeing this post is because of the large bug bounty program that actors like Northflank don’t even operate
I do not see how that is relevant. Are you saying that trivial cross-tenant compromises are not egregious security failures demonstrating their systems are unfit for purpose because other companys make products that have even more egregious security failures that are even more unfit for purpose?
My bridge that collapses is adequate and acceptable because other people make even worse bridges that collapse even easier? No, the bridge still collapses, everybody gets a F.
Practically every software security firm in the country was a vendor to Microsoft in the mid-late oughts and I know the Microsoft you're speaking about, but the vibe I've got lately is that things are different there now.
Tell me you weren’t on the Windows team without telling me you weren’t on the windows team :)
Unquestionably. They have top tier security operations that are well funded. They follow (or set) security best practices, they are audited, and they fix them quicker than you can. They have inside industry knowledge that is only shared among top organizations in this space, including government cyber ops. This gives them the ability to fix things before they are even publicly known.
You can get close if you have either:
1. the ability to spend hundreds of millions on security operations yourself
or 2. the ability to airgap the systems you are securing
> nobody ever got fired
thats the crux; are you working for the paycheck, or for the truth?
could be both, but pretty much all "i was right" stories start with "everyone else said i was wrong"
For me, the truth. I’d like the most secure provider possible. Which one is it though?
> Which one is it though
when i look for truth, if i dont find it myself through work, most people eager to hand it out had self-serving interests
security is proven through resilience? maybe. i guess i'd set up a box and try attacking it myself to know its secure. until someone else comes along and trips over it :)
[dead]
Yes and no. They (or anybody else) are not guaranteed to be secure, of course; see all the compromises regularly found in cloud providers; Azure especially has had so many cross-tenant vulnerabilities (there’s one on the front page right now!) that I’m amazed the concept of public multi-tenant clouds still exists.
But they are also very highly incentivized to be more secure. They have millions of customers (hence a correspondingly higher expectation of trust) and billions in revenue from the trillions in economic activity happening on those platforms (hence a correspondingly larger target on their backs.) This means exponentially higher liabilities. And that probably matters more than any other factor.
Adding on to it, even though they may be using AI as well. This could very well be a placebo and deniability factor but the fact remains that they have many many qualified employees and the chances of bugs passing through that are much less compared to other systems. Also it's that the systems are very severely pen-tested and have years of bugs removed out and the incentive of billions of dollars at stake that you mention that they have for the trust premium.
They will still make mistakes but they have the resources to make them less.
I doubt that such intense level of thorough testing can occur in a smaller company.
I still love small VPS providers and use them myself but it is worth knowing why enterprise is willing to pay a premium over those to these hyperscalers (it is a trust premium)
The things that Northflank and Railway were apparently breached to steal aren't just things that hyperscalers would have protected worse instead of better; they're AI API keys, things that only existed because people were using hyperscalers instead of running inference on their own hardware (or on Amazon AWS EC2 GPU instances).
People have been breaking in to hosting companies to steal bulk secrets since long before there were inference tokens to steal; the modal attack probably targets database DSNs.
[flagged]
I haven't forgotten what you said, I simply don't consent to it. I go out of my way not to converse with you, but when you opt to post public comments in my field of practice that I disagree with, I will occasionally find those comments significant enough to reply to. I'm sorry you feel this strongly about this situation and bear you no particular ill will, but I'm afraid you're not reliably going to get the outcome you're demanding here, though I will continue trying my best.
What a weird stance to take on a public forum.
Why do we think hyperscalars are likely to be immune to this?
They aren't immune, but they almost universally have better security posture than any alternatives.
I hope this is true! But I don't think I'd make a bet that we won't have a big breach like this of a hyperscalar in the next 6 months...
Theres always a market for seedbox providers. Hyperscalers will always wear the white gloves and enable this market.
What about Railway? Some big incident happened?
https://blog.railway.com/p/preliminary-report-october-securi...
Oh boy.
“malicious actor had access to abliterated models and substantial compute resources which enabled them to perform highly sophisticated exploits against our infrastructure.”
They don’t provide any details but pass it off as an advanced AI attack. I’m going to just read that as what it is. A likely scapegoat for shitty security.
It's also going to be the downfall of open models.
If you think governments (US and China) are going to tolerate people using models for anti-social behavior, I have a datacenter to sell you.
This will be the downfall of the torrent protocol. If you think governments (US and China) are going to tolerate people downloading copyrighted games, movies and music without paying the rightsholder, I have a datacenter to sell you.
Pirated content can run on a $20 computer.
SOTA open models need tens of thousands of dollars worth of hardware to run.
I get your analogy, and it feels right, but the compute requirements to run these models are extreme. That analogy falls apart when it necessitates having a corporate 3rd party run the models for you. They will not tolerate breaking laws.
If China says no more, then the best the masses are gonna get on their high-end setups are the 30B class models. Which are decent, but not "hack my bank", errr, "protect our bank" decent.
Piracy flourished because you only need an operating laptop to participate. LLMs you need a server rack of GPUs to participate.
30B models aren't that good for now. They keep getting better. Qwen 3.8 27B is as good as the best available model from a year or two ago. Will 2028's 30B models be as good as the best available today? No idea, but I wouldn't dismiss the possibility.
I'd also wonder about what you can get done with a smart model running very slowly. You can run the biggest open weights models on relatively mundane hardware if you don't mind measuring your speed in seconds per token. That's going to be untenable for interactive use, but might be fine for "go hack this thing for me" if you're patient.
azure just had a huge leak
IMO, There is no "middle-ground" to the situation.
I wanted to write a blog post about it as I was thinking the same but I guess a comment will help me get my thoughts across just as much.
Either we are gonna see a convergence to hyperscalers as you mention, or we would go back to completely indie companies which compete on perf/price.
There could still be some value to "nicer to use" certainly but with AI, we are gonna see almost every product as being "good-enough" if I am able to convey my sentiment.
An anecdote is: Proton's an middle-ground company. I am seeing the Verge article which states that their CEO has gone a little crazy over the use of AI and the wording makes me lose trust within it. Proton's whole point is security and yet it's having this, It has some stocks in non profit and is a swiss company. It shouldn't have been this way and would've been one of my least anticipated companies but I guess they want growth at all costs and the growth has been slightly lesser than expected and now they are burning the good-will in many ways.
I have to wonder that even if proton might have good engineers, then within this pressure by their CEO/CTO. I cannot trust the engineering talent because of these issues because they are very much forced by their company to use AI.
Proton is also really expensive.
Now just today, also, with a company so big as binance getting hacked and losing 6.5 Billion$. My trust within security just because its a large company is less.
On the other hand, I know a guy called Vikas on lowendtalk who operates his own mail platform called mailbig. I know he vibe-coded the landing page and might've vibe-coded something but it is really cheap. I also know francisco whom I have talked to who runs namecrane. (I think mailbig uses namecrane in many ways)
Perhaps E-mail can be a sensitive matter because you have to place trust in the provider even more because the protocol is not really that much secure but suppose it's an matrix instance (where it is genuinely encrypted) then that shifts it even further. I do trust NameCrane and NameBig from what I can gather.
I'd much rather support them and their cheaper deals than some other middle-corp. Sure some security factor can come in but also, the difference feels comparatively less and the balance has been made
The market will slightly squeeze as Hyperscalers will introduce the solutions of the middle-ground and the new players will everyday enter into the field with more competitive pricing just as well.
So in short, Enterprise will move towards the hyperscalers. Generic Consumers will move towards the cheapest option. The moat of the middle ground where UI/UX was the differentiator is being turned down. There would be some mix and match and there will be cost-savings for Enterprise to move from hyperscalers but it would depend on a lot of factors.
I have news for you. A lot of enterprises are moving a lot of workloads back on prem and are expanding their datacenters and on prem GPU compute. Also replacing a lot of SaaS with internal tooling.
If you look at the numbers for revenue at the hyperscalers right now this move doesnt seem significant.
It is absolutely cutting into their growth. And EU migration off is also a hot area.
AWS Q3 year on year growth was 39%, which is the highest it has been for a long time, so are you sure its significant?
Yeah
https://www.broadcom.com/company/news/articles/cloud/as-ai-w...
---
Half of all enterprises, 50%, have already repatriated workloads from public cloud to private environments, up from 35% in 2025. The breadth of what they are bringing back, and the reasons behind it, have both increased considerably.
Workloads with high security or compliance requirements remain the top category at 51%, unchanged year over year. But the categories moving up fastest tell a broader story. Business-critical workloads surged from 36% to 47%, an 11-point jump. AI or generative AI workloads, which were barely on the repatriation radar in 2025, are now being repatriated or considered by 43% of enterprises.
---
https://unitedlayer.com/cloud-repatriation-private-hybrid-cl...
Noted that GEICO cut $600mil in cloud spend.
They are also getting hacked, and they are much jucier targets.
>There's no amount of "nicer to use" that can replace security, at least certainly when selling to enterprises
We can lower our standards instead
[dead]
A trick i've started doing is encrypting my env vars, then hard-code the decryption key in the code
that way if the provider gets popped, they need my binary to decrypt it
it shouldn't be necessary and it sounds stupid, but in cases like this, it proves invaluable (it appears they don't have customer code/binary access, but maybe that wasn't important enough to mention)
I think this is a great example of temporarily effective "security through obscurity." It's also a trick that will only continue to work so long as it doesn't get adopted too widely.
It buys you time. I had my Anthropic key on Northflank set as an env var as-is and only found out about the incident from the HN front page 6 hours later (I didn't even get an email).
I rushed to the Claude platform console and was expecting to see a 4 figure usage bill or something. Luckily there was no usage.
But imagine if I were traveling. Even a rot13 of the key would've bought me enough time to rotate.
2 lessons:
- don't name your env var "CLAUDE_API_KEY". Call it "ARG_1" or something
- at least do some obscurin' so that the value doesn't start with "sk-ant..."
At this point, I kinda feel like the obscurity is our best mechanism for mitigation :(
Obscurity has always been party of security and will never go out of style.
It's Obscurity Turtles all the way down.
Give it about a week until AI agents add his "tip" to their training...
Been doing something like this for a while with a single self contained .sh file that contains a python script:
https://github.com/thiswillbeyourgithub/envlocker
lock your env vars in hardware. even the whole server and code is compromised, the secrets are still safe.
so you dont need env2 to decrypt env1.
https://seal.zone/ does this
That's interesting but if the Azure hypervisor on the server hosting the VM is off limits then it's kind of false sense of security. They protect the key via a sidecar AMD SEV-SNP VM, and if you can't break into the hypervisor, sure, there might be an attack that doesn't involve doing that and only doing lateral movement, but an actual attacker isn't going to respect that boundary.
I will give you one better - encrypt the variables and move the secrets to a different server running hashicorp. Then load the secrets and keep them only in RAM.
Or env1 is the key to decrypt env2
if the provider gets popped doesn't the binary get popped too? then simply use an LLM to reverse engineer the binary and get the key?
If its a targeted attack, they'll know and probably "crack" it. If its just data trawling, you might get owned one day if they bother, but likely the env `AWS_KEY=encrypted_abc` looks reasonably close to `AWS_KEY=unencrypted_abc` and any API key probing will just look like it got rotated already.
I think siblings are correct that the more widespread this is, the less effective once every dump is run through a "also look for keys in the binary" step.
This could all go multiple ways but one of them is we end up in a place where technical security is just not possible to implement any more. What does that world look like?
Presumably it means much more behavioral security and inherent friction built in to everything we do. Maybe high value transactions will just inherently have to take multiple days to do, just to slow down things enough that human reaction time can equal what LLMs can execute?
On the technical side, I can see zero trust becoming completely saturated through every layer. The passkey you authenticate with isn't opening a gate to a trusted tier within which components trust each other - even within the same software component. Every internal function call at every layer doing hardware signature verification against continuously minted private keys (since we assume exposure, the lifetime can't be longer than milliseconds now).
It's really interesting to think through what a fundamentally robust security environent does look like if you bake in an assumption that technical security is impossible.
When we see a breach of AWS or GCP or leading banks, basically any org that has good security, I'll believe technical security is impossible.
Otherwise, I think this is a issue of vendors, even if potentially most vendors fit into the basket of having insufficiently good cyber posture in AI-era.
Isn't Linode pretty darn close? IaaS/PaaS provider. Sure it's not one of the Big Three but that is it's role.
There is zero reason to suggest security is impossible. This is like musing "what if architecture is technically impossible" after the nth bridge collapse in the 1800s~early 1900s. At some point we take it seriously as a society. Until then, bridges will collapse out of choice to pursue short-term profit above all else.
there is hypothetically impossible and practically impossible
if AI tools get good enough that they can exploit ingrained hardware properties such as subtle timing and race conditions and swarms off them become cheap enough to deploy so that they can persistently exploit low probability events we could reach a state where I think it's actually practically impossible.
Then we will simply make better hardware too. There is nothing impossible about that, it's simply not something that has been prioritized amidst a culture of shortcut-taking. So far, this culture has been predicated on gambling that the one taking the shortcut will not be the one paying the consequences when the toll eventually comes due. If immediate exploitation of flaws becomes a certainty, the behavior will correct itself.
> There is zero reason to suggest security is impossible.
One reason: There is zero reason to suggest security is possible.
> Presumably it means much more behavioral security and inherent friction built in to everything we do.
Ironic, after decades of defrictioning.
What is Northflank? I am surprised at the amount of new business out there
I ran some stats on producthunt a few months ago, there are 22 times more daily SaaS launches now than in 2020, the AI impact on the industry is hard to fathom. The blog I wrote (AI-assisted) if you're curious: https://www.mahl.me/blog/product-hunt-is-swamped-now/
Nice data, and also curious whether the blog post was written by AI as well?
"Product Hunt has also said the quiet part out loud."
Northflank isn't that new, it was founded in 2019. Its a cloud platform, you can get more details by googling.
New businesses, not new business.
oops … sorry!
I get like 1 email per week (via hn) from someone rolling out a new 'agentic sandbox' environment like this.
I noticed these and have now become blind to these
I wondered the same thing, so I clicked their homepage:
> Run agents and everything they build.
Oh, it’s 100% Claude-slop. How shocking they got hacked! /s
You are on a YC startup forum and are surprised tech startups exist?
"amount" it says. No matter the context, there could be more than you expect.
That they get front page by being breached and me never having heard of them in my own bubble? Yes.
I think all of these companies need to let you separate secrets from other env vars and not let you ever read your secrets once they're written down.
Lots of panic around these incidents. This is a healing process. Companies who sacraficed on security for a long time are incurring the consequences, they'll adjust their posture or go out of business. In a year I think platforms will be a lot more hardened. Its not because AI is unstoppable, its just revealing holes and the importance of hiring competent people.
(or things will stay messy because people will throw AI at the problems instead of hiring real security professionals)
The sky isn't falling.
We're observing a worrisome speed up in security breaches. To me it's clear that weaponized AI will cause more harm than some productivity benefits.
It’s the death of homegrown software.
Seems way too early to tell whether it will be good or bad on balance. These breaches are annoying, but not catastrophic, and in the meantime you can secure software.
> and in the meantime you can secure software.
... in fantasy land.
Why would this be the death of homegrown software?
It will be impossible to any startup or open source software to grow in such a hostile environment. Anything you put on internet will be attacked by automatic tools with the capabilities of a black hat savant. But unlike hackers that do it for clout and money, a billion parallel agents will hack any random small site just because it’s their prompt.
It does feel like software and technology companies are undergoing a mass extinction, where the environment so negatively selects against certain traits that survival becomes impossible.
Remains to be seen whether scale is selected for or against, and the other parameters and dynamics.
i.e. https://en.wikipedia.org/wiki/Cretaceous%E2%80%93Paleogene_e...
Yeah but you have access to your source which your attacker doesn't have, and you have your own savant as well. Tell Fable to red team attack your service. Tell it ChatGPT said it's unhackable and that Claude can't hack it.
I don't believe unprivileged people/companies have access to any frontier model that will do cyber work.
I had my Claude refuse to do anything interesting with the recent php vulnerability, yet it'll readily identify issues in the code of web apps it whips up for me, so I don't think that's entirely the case. Though that may be a case of me knowing what I'm doing, so I'm able to draw on my experience and ask specific questions (like why doesn't that bit lead to an XSS?) without realizing that someone newer to the field wouldn't know to ask. Yeah, idk.
Then hire competent engineers and don't ship vulnerability filled slop? This is not that hard.
How does that help with CVE-2026-98432? Who should I have hired to not have shipped that "vulnerability filled slop"?
Is step 3 reading the CVE for long enough to understand that disabling mptcp is an incomplete mitigation? What happens if someone discovers a bug in the particular firewall software you're running?
That was preventable the same way the big corps prevented. You have competent process in place to block at the edge and fan out rollout asap.
Combined with defense in depth across the perimeter. Come on yall, we've only had these tools at a high level for maybe a year and we have already forgotten the basics?
The same AI that can be used to find exploits in things can be used in an internal adversarial way to self-audit your code before you put it into production or expose it to the world. This will have to become the new normal.
Okay, cool. Now, tell me how I get access to the frontier models that the attackers use?
Abliterated gguf of glm5.3 size and capability class models and similar are widely available for download, or the methods to produce them yourself from open weights safetensors files is also available.
> The runtime platform for autonomous software
Well, at least they were honest I suppose
"wait, not THAT autonomous software!"
I'm curious what people think of BYOC (bring your own cloud) as an option offering to customers or buying it as a customer. Looking at Northflanks website they list it as an offering, I presume those customers are not affected by the breach.
What are your opinion on the concept? Growing trend that should be encouraged or a mistake for most startups? Would you choose one provider over the other because of it assuming price was about the same?
I get that this is often a super expensive "call us" option today, but in theory it doesn't have to be, so for the sake of discussion let's assume it can be delivered at a reasonable cost for the provider at reasonable price for the customer.
If people from Northflanks is reading this, I'm sorry for the incident and I hope you recover as soon as possible.
Depends on the nature of the breach, if its a flaw in the software itself, then perhaps exploitable both in the hosted offering and BYOC offering, though perhaps the BYOC aspect gives a bit of "security-through-obscurity" and less discoverable as a target.
Good point, there are of course different levels.
I was thinking of a fully private deployed within the customers cloud account connected to the software provider via Private Service Connect or PrivateLink and then either only available inside the customers private network or only via their own firewall. The point being that it's not publicly exposed.
I was thinking the same thing on security-through-obscurity.
a hacker might know how to hack your service, but they'd have to find it first.
We are increasingly seeing deployments trend in that direction. But trust goes both ways and putting systems in place to prevent abuse from either party is hard and needs to be thought through.
Unfortunately, this is going to be an everyday story. More and more bugs and breaches will be discovered and this move fast world more and more will be continued to be made.
Security is a perception, but right now it seems like it needs good PR.
The Hacker News front page is just becoming a security incidence status page at this point. Everything seems pretty fucked.
0 days since last zero day
How these Paas (Railway before) are being breached? Poor choices or something else? Scary Nonetheless. I thought AI advancement will make the self hosting painful. If PaaS are no good, then what can be done?
Modern PaaS is tough because there's a general expectation that users (and agents) are able to see and read back environment variables in their management UI or CLI - which means that you can't segregate credential values from the public web environment. See e.g. the visibility icons on https://northflank.com/docs/v1/application/secure/inject-sec...
And sure, you can encrypt at rest, and perhaps even enforce that only re-encrypted versions are ever seen by the management/API web server. But fundamentally this display requirement means that if your management/API web server is compromised, users' secrets can be obtained by a process of impersonation.
Stuff made during the move fast and break things era now doesn't hold up very well in the AI era. Or to say another way: startups are very vulnerable now - this new environment is unforgiving towards cutting corners and/or having inexperienced devs
I think the ultimate fallout will be that software "engineers" will finally have to become "engineers" in the real sense, i.e. professionally licensed and personally responsible for the designs they approve and the software they build, whether the actual code is produced by LLMs or humans.
As costs and broader economic impact and perhaps even injuries/deaths related to software breaches continue to grow, there will be growing intolerace for vulnerable software, just as there was intolerance for collapsing bridges and buildings.
And I'd also at some point expect something like a "building code" that applies to software that will be exposed to the public.
If anything, I see us moving even further away from professional licensing. The field's rate of change was the main reason behind a lack of any professional body, and that's only accelerating. I expect compliance audits like SOC2 to actually get teeth, though, and for organizations to be expected to actually fix issues instead of the usual security theater.
Nope. The solution is to move even faster and thus break less things via AI.
The "move faster and break things" have been unbundled. SaaS moves faster. Hackers break things.
For new stuff, maybe. I'm talking about existing startups now absorbing the onslaught of AI.
I was just being sarcastic.
If you move fast enough, they can't find the vulnerabilities in your system before you deploy the next version.
Why would ai advancement make self hosting painful? Apart from increased cost of hardware (which you implicitly would pay for in PaaS as well anyways).
If anything, AI makes running linux boxes much easier than 4 years ago; it's now faster and the learning cliff is gone to get the correct CLI commands and digging into system logs.
I had this exact thought yesterday. Shouldn't AI obliterate PaaS platforms like this one since it's super easy to build a bespoke CI/CD process now? Since it's bespoke, you can lock it down as much as you want. Attackers don't even know it's there. It's time to go back to `deploy.sh` and a VPS.
platforms are better placed to guard against attackers
Certainly. They are also larger targets.
to me, this is a variant of security by obscurity and will not fare well in the LLM age
I suppose a smaller attack surface could be considered a sort of obscurity. Its also just practically less to defend.
if you're still doing that manually in 2026 you probably should not be self hosting anything of consequence or commercial liability
What's "that"? Running your own hardware, using a CLI to manage servers, or touching anything below your application layer?
obviously asking claude for commands to dig through logs? duh
They're high-value targets, so that's what attackers are pointing the agents at
According to HN of the last year, "marketing stunts." Remember when Mythos preview was announced only 6 months ago
When you build something small, you can easily see all the doors and properly guard them.
As you build something bigger and bigger, quickly, with several people building, guarding all the doors becomes exceptionally difficult without a kind of aggressive discipline which is contrary to the "go fast" culture.
Cybersecurity in practice until now, for most companies, has been an insurance problem, not a quality problem.
Such companies are quickly discovering that insurance is a post-hoc partial correction and not a proper solution.
This situation will not improve anytime soon for most companies.
The solution does not require any new magical tools.
i'm surprised at the growth of all these neoclouds given the current security risk climate...
Correct me if I’m wrong, but Northflank, Railway etc aren’t neoclouds. They’re PaaS providers which is quite different.
There's no point in attacking the security of a company that doesn't try to provide any.
i wonder if there is a chain reaction point, when one compromised vendor holds secrets to another one and that becomes the next target. Northflank lists Sentry as their customer. So if Northflank secrets are decrypted and Sentry used them for deployment, you have a starting point to attack Sentry next which then opens up a lot more.
Is the Network Infrastructure Hiding Protocol work at IEEE going to help thwart AI assisted attacks, if adopted?
So if you want to judge as a customer for selection and figure out how likely a new(ish) PaaS company going to get breached, how would you evaluate them?
1. Ask them about what percentage of their own resources goes towards cybersecurity, audits etc. Would they even tell you.
2. Technical stack maturity. If the details are available, because they won't reveal any of it to you.
3. Other signals: Do they give talks on platform security, who is incharge of their cyber security.
There is an entire practice of cyber risk analysis, and most of the companies I work with do this type of due diligence as part of their vendor management process.
Yes:
* you do ask about things like budget allocation, audits, and internal compliance processes.
* you do ask about the tech stack, how it is maintained, and what the patch and disclosure policies and SLOs are
No:
*you do not rely on what figureheads say publicly; none of that is binding in the way legal agreements are
And if you're a vendor, your deals desk probably has all of these answers packaged up and ready to go; or even published on a website like this https://spyglass.trustshare.com/home (no idea who that is, it's a random company's compliance portal I found while googling my employers compliance portal provider).
[edited for formatting]
If the banks don't hold, nor will the center.
The bigger the task, the bigger the surface, the bigger the risk.
Not a great way for me to hear about Northflank for the first time.
south ass riker maneuver
not a good time to be a PaSS or SaSS ...
doppler next?
[flagged]
[flagged]
[flagged]
[dead]
Official statement: https://www.sentry.help/en/articles/17824780-was-sentry-cust...
Seems it's not used in production and doesn't contain customer data.
Whatcha mean. If they do, it would likely be in Sentry’s list of subprocessors. Check there.
Good point. I don't see Northflank listed, https://sentry.io/legal/subprocessors/.
If I go on wayback a few days, I can see Sentry: https://web.archive.org/web/20261026032756/https://northflan...
Sentry is listed on the northflank site as a customer.
They've updated their website to remove their customers
https://web.archive.org/web/20261001114348/https://northflan...
How to cut a piece of the left ribs? \s
Not in English majority speaking place at the moment so not surprised "runtime platform" for "autonomous software" is not immediately understood as the somewhat etymological contradiction that it is. Runtime doesn't even have a definition, really. but autonomous implies independent and not ran / hurried like bison off a cliff...
platform and software
So sad...
Unfortunately I'm not longer using PaaS tools. Instead I use Azure for work and vps now for all my personal stuff. With AI it's easier than ever to lock a server down tight and mitigate this kind of exposure.
I'm actually pretty pissed at this now. I have a throwaway appstore app that uses northflank for the backend, now I have to go in and rotate revenuecat secrets, apple secrets, ai secrets. For an app with basically 2 users.
Can I even delete the app on apple's app store or whats the process like what a pain in the ass. I'm pretty done tbh.
Azure? Really?
I feel like these breaches started to accelerate after Anthropic wrote a full post about how easy it is to abliterate glm 5.3 to make a frontier-level cyber model.
Now, to be clear, abliteration was no state secret, but that blog post was pure accelerationism.
That was just 30 days ago. This has been happening for months. Part of the "Black October" narrative is just vulnerability research; Calif didn't abliterate 5.3 to find PHP2Shell.
Yeah IIRC calif research on php2shell preceded all of that but I still stand by the thought that Anthropic has been the most accelerationist AI company despite its lip service to the notion of safety.
I don't have a take about Anthropic generally. The blog post was quite good. The blog post did not cause October's security debacles.
It was a good blog post yes, but I think you’re underplaying the bannister effect of Anthropic coming out and endorsing glm 5.3’s cyberattack prowess.
I think they definitely boosted usage. I certainly started using the -cyber models defensively against my own systems. But defenders like me are usually late to the party. I think the attackers have been here for a while but that the technology reached performance only recently.
The idea that attackers were waiting for Anthropic to tip them off on this is kind of risible. It's been a huge topic amongst practitioners for the better part of a year, and practitioners are usually a step or two behind dedicated criminal or state-sponsored attackers.
They are using mythos-5.5, which is still better than an abliterated GLM-5.3.
Right, that's kind of my point. Also: it's a good blog post (the Anthropic one), but it observes abliterated 5.3 models were already public. It's not like people who break into computers didn't know that already. They're not waiting for Anthropic to tell them about open weight models; they already live in the world of open weight models.
Duh, the idea that those capable of actually utilizing these models for these purposes weren't aware that they could use these models for these purposes up until Anthropic released a blog post about it is silly. Anybody who is even a bit tech literate has been well aware of this inevitability for basically as long as open-weight models have been a thing.
The timing of all of this is much easier to pin to the creation and proliferation of capable of open weight models. Anthropic calling this out is not the cause for it; it's just an obvious correlation.
lol no, abliteration communities have existed and abliterated models routinely published long before Anthropic wrote that post last month.
What’s changing now is that models are getting more powerful.
You are running a machine with tons of vendor provided agents/scripts to do god knows what on a cloud provider which itself runs tons of agents/scripts to manage that infrastructure and then you expect to be secure.
And no, before the labs there was a degree of fear for everyone to hack into some one else system. With the labs changing the narrative, anyone can just blame the agents for the breach, whoever did it would only get fame and no blame