Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.
Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or that unofficial github status page repeatedly posted here that counts all github services together into one number.
I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach.
Smaller is, in fact, better, because it's easier to operate reliably.
What I'm curious about is why it is a single-PaaS; I'd have expected Salesforce to have the customers quite isolated so the chance of bringing down multiple customers at once was much smaller.
This is the case for every single B2B saas product. This is like the "bar is rolling on the floor" level of competence required. Please have higher standards for paid products.
I don't know, that reads exactly like an AI troubleshooter working through a plan without the implicit contextual understanding an experienced human might bring to either the actions or the communications.
"Oops, we forgot to tell it that this is the hyperscaled Salesforce production environment and that its choices need to project competence and consider brand embarrassment. WILLFIX"
In my experience it’s a safe way to do something useful while everyone is getting their bearings. It immediately partitions the situation space between being persisted or systemic vs local or caused by long-running processes. Plus everyone’s going to ask if you’ve tried that already, so you might as well get it out of the way if it makes any amount of sense
This is what happens when more than half the company is away attending the Salesforce cult-indoctrination stuff while spending all their bandwidth making customers/partners feel good.... The stuff that matters to keep the lights on gets overlooked.
arguably, this is what sales cares about and the half-measures taken to tackle what must be the Mt. Everest of tech debt at Salesforce is what leads to large, systemically degraded customer trust in products that keep shipping bugs
I don’t think engineering and SRE of the organizing company are ever invited to those events. They’re mainly for marketing and sales (which includes solution architects).
Cause: Legacy Salesforce login service got into a resource-exhaustion cascade.
Fix: Rolling some unspecified fix they proved in testing out over the fleet seemingly very slowly (After their earlier attempts to roll something out faster failed).
It lists all instances, you can drill down into each one of them, see which services are affected, and each affected service pops up the incident timeline?
Isn't it actually amazing, and not "the most salesforce thing ever"?
You can click on any of the instances and then the service that is down to read the updates. It’s not 100% clear but some sort of issue with a “legacy login service”. The latest updates say a fix is rolling out.
As someone who has never used Salesforce nor hubspot. What are the core features that has people using these services? Is it just the integration between all the different areas where customer data lives?
It’s like a lot of these big SaaS platforms that lock people in to big contracts. They promise the world with all sorts of features and integrations but most customers don’t use that and get locked into buying a very expensive solution to a very simple problem (in this case tracking sales pipelines).
Where customers do use the features and integrations it’s often a giant mess that needs a whole separate ecosystem of consultants and “partners” to get the thing working and maintaining it.
Peak Tech Salesforce was 2010 +/- 2 years - i.e. after Visualforce and before Aura era. It used to be a developer oriented platform and it became shiny/flashy garbage eventually. But all these shiny things allowed them to get a large market cap with very brilliant sales people, it's hard to deny.
My gut instinct is that this is about when all of their on prem servers were EOL and their /public cloud solution was required. This must have had something to do with that
And nothing of value was lost. God, I hate everything about Salesforce. Sometimes I have to integrate against their services, and it is always a pain, not to mention what the core project actually is: optimization of marketing and spam.
Could be. With a big conference going on and the updates mentioning resource exhaustion it could be a bunch of people doing demos, AI driven or not. Basically slashdotting themselves.
"And in tonight's news, the worldwide CRM solution Salesforce had a global outage affecting one hundred percent of its customer base. We interviewed users of the service to find out the scope of the impact. Everyone agreed that they were impacted, but strangely, nobody could describe _in what way_ they were affected."
Kind of ironic. Salesforce is basically one of the major spiritual grandfathers of Slop. It is not uncommon in production systems to find that objects like Contact and Account have hundreds of custom fields. Sometimes, you find out that several of them have the same meaning and semantics, but were used at different times. Digging out you discover that some Marketing guy that used to work at the company did some task in a certain way that was lost when he was gone, and then a few months later his substitute had the same need and went ahead and created the same field with a slightly different name.
Doing data engineering work with Salesforce data is an exercise on archeology, psychology and organizational politics.
Slop is basically the ontological and teleological philosophy behind Salesforce very existence. Despite the official discourse that the "No Software" meant no infrastructure, no toil with updates and configuration, the subtext as intended for executives was very clear: "No need for you to be blocked by those pricks from engineering and their stupid, bureaucratic and gatekeeping rules".
"No software" was a call-to-arms to a certain subset of managers that were radicalized by Nicholas Carr's 2023 HBR article "IT Doesn't matter". It doesn't matter that Carr was a journalist and a writer with a masters in English that has never ever run even a small bodega, or has never managed an IT department. Anti-intellectualism and the abundance of capital brought in by the petrodollar that allowed the US government to run deficits year by year while exporting the ensuing inflationary effects to rest of world, would ensure that this message would ressonate and then even be amplified during the years of ZIRP and the Baillouts. Play fast and loose, first come, first served, a rising tide rises all boats and all that jazz. Wall Street favors bold, and the heck with the long term! This quarter will only live once!
Frankly, this is just poetic justice: Kill by slop, be killed by slop.
Typo: Carr wrote that in 2003, not 2023 for those of you who missed the foolishness and insane wreckage that article caused.
The lost business value and competitiveness caused by outsourcing IT overseas to unmotivated parties under Carr's premise is hard to put your finger on but I have seen the aftermath and it's pretty massive.
Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.
Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or that unofficial github status page repeatedly posted here that counts all github services together into one number.
> which belies deep misunderstanding
I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach.
Smaller is, in fact, better, because it's easier to operate reliably.
What I'm curious about is why it is a single-PaaS; I'd have expected Salesforce to have the customers quite isolated so the chance of bringing down multiple customers at once was much smaller.
a few things are global (login service) to some extent, just as AWS places several such things in us-east-1
Hmm, could the use of genAI have anything to do with this failure and the inability to quickly fix it?
I honestly don’t get the snark. The status page has:
Seemingly meaningful IDs
Search
Region filter
Email update signup
Predictable URLs for instance status so they can be deep linked in runbooks
What appears to be the actual live instance status.
What appears to be the actual live service status in each instance.
An update log with frequent detailed updates.
This is the case for every single B2B saas product. This is like the "bar is rolling on the floor" level of competence required. Please have higher standards for paid products.
What do you think of Atlassian?
Have you tried turning it off and then on again?
> We're no longer pursuing restarts as a path to remediation.
Oh you have
Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it.
> We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue.
At least it didn't fix the problem so they can actually start finding the real cause.
> We're no longer pursuing restarts as a path to remediation.
Why isn't the AI they sell telling them what's wrong? Why do they need to take shots in the dark to "see if that resolves the issue"?
I don't know, that reads exactly like an AI troubleshooter working through a plan without the implicit contextual understanding an experienced human might bring to either the actions or the communications.
"Oops, we forgot to tell it that this is the hyperscaled Salesforce production environment and that its choices need to project competence and consider brand embarrassment. WILLFIX"
In my experience it’s a safe way to do something useful while everyone is getting their bearings. It immediately partitions the situation space between being persisted or systemic vs local or caused by long-running processes. Plus everyone’s going to ask if you’ve tried that already, so you might as well get it out of the way if it makes any amount of sense
Followed quickly by "Redeploying with more log lines", the next logical step.
Been many an MIR that I've seen where after initial assessment, the next log entry was "service restart attempted"
Restart should be a very last emergency step, as if it works, a restart often might wipe out evidence of why.
So hopefully it's not done often.
"Yeah, I'm with Rob. Just let's reboot and see what happens"
"If that doesn't work, clear the cache and reboot again."
Unplanned outage timing is never good but this is really not good.
https://www.salesforce.com/dreamforce/
Sept 15-17
Say what you want about the product and leadership... they do throw a good conference.
Probably not a coincidence
Yup. Could be that everyone was rushing to get all their products and demos ready leading up to it.
This is what happens when more than half the company is away attending the Salesforce cult-indoctrination stuff while spending all their bandwidth making customers/partners feel good.... The stuff that matters to keep the lights on gets overlooked.
Arguably, making customers and partners feel good is the more important part of the business
They wont feel good if the product they pay for doesnt work
From what I can tell, the business model is "it doesn't work, but you can pay folks exorbitant fees to 'customize' it for you"...
Also see: Oracle
arguably, this is what sales cares about and the half-measures taken to tackle what must be the Mt. Everest of tech debt at Salesforce is what leads to large, systemically degraded customer trust in products that keep shipping bugs
I don’t think engineering and SRE of the organizing company are ever invited to those events. They’re mainly for marketing and sales (which includes solution architects).
this event is only for customers. its not a company event.
Cause: Legacy Salesforce login service got into a resource-exhaustion cascade.
Fix: Rolling some unspecified fix they proved in testing out over the fleet seemingly very slowly (After their earlier attempts to roll something out faster failed).
Details at https://status.salesforce.com/incidents/20004433
I wonder if “legacy login” is the shared login gateway.
It’s optional but everyone uses it. And it was flaky for an hour or so, like two months ago.
That status page is the most salesforce thing ever.
Scroll down. >_<
Wow, it's almost as long as the Every UUID V4 or Every Floating Point Number pages.
I see tabs with a spinner loading infinitely, which is indeed very salesforce like
It lists all instances, you can drill down into each one of them, see which services are affected, and each affected service pops up the incident timeline?
Isn't it actually amazing, and not "the most salesforce thing ever"?
At least they're consistent about UX. My only complaint is that it needs more tabs.
Yup. No mention of outage. Even drilling down gets nothing more than "Service Disruption".
You can click on any of the instances and then the service that is down to read the updates. It’s not 100% clear but some sort of issue with a “legacy login service”. The latest updates say a fix is rolling out.
At this point OpenAI really ought to let us know when they're testing again.
My thoughts exactly.
You better bet someone started their agents with a prompt "Make a salesforce clone but with 100% uptime"
"Make a salesforce clone but with 100% uptime"
Salesforce has turned into the monolithic messy bloatware it set out to replace.
Please VCs stop with the AI FOMO and find a few good startups to just go destroy Salesforce and give folks a simple inexpensive replacement.
There are 10,000 “simple inexpensive replacements”. Always have been. If you need a glorified three object contact list… you shouldn’t buy Salesforce.
Those just, obviously, can’t do almost any of the forty million serious things that Salesforce does, and real businesses do need.
As someone who has never used Salesforce nor hubspot. What are the core features that has people using these services? Is it just the integration between all the different areas where customer data lives?
It’s like a lot of these big SaaS platforms that lock people in to big contracts. They promise the world with all sorts of features and integrations but most customers don’t use that and get locked into buying a very expensive solution to a very simple problem (in this case tracking sales pipelines).
Where customers do use the features and integrations it’s often a giant mess that needs a whole separate ecosystem of consultants and “partners” to get the thing working and maintaining it.
I can't understand how such a huge company can have such a lousy UX.
Peak Tech Salesforce was 2010 +/- 2 years - i.e. after Visualforce and before Aura era. It used to be a developer oriented platform and it became shiny/flashy garbage eventually. But all these shiny things allowed them to get a large market cap with very brilliant sales people, it's hard to deny.
Is like Microsoft Windows in that way?
This is certainly a unique status page.
It's dns isn't it
Feel like it has to be for all of this to go down at the same time.
I was scrolling to see this comment, haha
It's always DNS.-
More like 70% human-configured DNS, 25% human-configured routing configuration, 5% interesting software bug.
Entirely correct.-
(Nowadays any of those need to fit in an "agent dropped all tables. Apologized" moment.-)
No, IPv6
Classic
Haven't they got some kind of new fancy ai interface they can use to fix it?
Perfect timing with Dreamforce this week.
Leetcode developers win again
Here is link to incident details: https://status.salesforce.com/incidents/20004433
My gut instinct is that this is about when all of their on prem servers were EOL and their /public cloud solution was required. This must have had something to do with that
My gut instinct is that this is about Dreamforce with the rickshaws and whatnot
I'm sure the cause of this outage will not be connected to vibe coding in any way
ClaudeForce in action!
Remind me please, what are folks currently paying per seat for this glorified CRUD app?
It's this thing called golf course driven development
Permanent cache for that one.
If I had been just out of Uni I would think this is edgy
Now I'm just glad I'm not responsible for this fire
Seems it is back up now. Damn.
More of this!
Dreamforce, wake up. You’ve overslept!
And nothing of value was lost. God, I hate everything about Salesforce. Sometimes I have to integrate against their services, and it is always a pain, not to mention what the core project actually is: optimization of marketing and spam.
Sure, if you’re struggling with weird custom endpoint behavior, or custom triggers and validation, that’s on your admins and devs.
But if you’re having trouble with the standard rest or bulk apis, that’s 100% on you.
What is painful about a plain REST API?
not really defending salesforce but OAuth+REST is a pain? Pretty plain vanilla in terms of integration requirements.
Does it mean I won’t get any AI generated slop spam for a few hours?
Could it be people doing Claude/GPT automations and they just can't handle it?
Could be. With a big conference going on and the updates mentioning resource exhaustion it could be a bunch of people doing demos, AI driven or not. Basically slashdotting themselves.
Wow, the intern must have tripped over a very big power cable this time
So I guess today everyone gets actual work done
Sam speaks at Salesforce.
Salesforce goes down.
No causation here...move on.
Huang, Sam and Dario speaking at Dreamforce.
Wasn't it Dario
Ah yes. Exactly what a status page should look like: an endless list of random ID’s that don’t mean anything and no information whatsoever
At least salesforce is consistent with their design language
> random ID’s that don’t mean anything and no information whatsoever
If you use salesforce you know what all of that stuff means. Just click on one, it’s not rocket surgery.
Was really just poking fun at them - AWS’s status page isn’t much better
Random Ids? If you mean the “USA324” ones, those are pods. If you’re a customer you know which one(s) you care about.
Looks like a region list to me, maybe just with a lot of regions
It’s akways DNS or login
Was it DNS? Any guesses? :)
Vibe-slop, push to Prod?
They let their agentic AI take over system maintenance at Dreamforce yesterday like they were pushing in the talks
/s (partly)
At least now we can figure out what Salesforce does.
"And in tonight's news, the worldwide CRM solution Salesforce had a global outage affecting one hundred percent of its customer base. We interviewed users of the service to find out the scope of the impact. Everyone agreed that they were impacted, but strangely, nobody could describe _in what way_ they were affected."
Obligatory https://xkcd.com/2347/
Never before in the history of global compute outages was so little lost by so many down servers, whose purpose was known to so few.
Kind of ironic. Salesforce is basically one of the major spiritual grandfathers of Slop. It is not uncommon in production systems to find that objects like Contact and Account have hundreds of custom fields. Sometimes, you find out that several of them have the same meaning and semantics, but were used at different times. Digging out you discover that some Marketing guy that used to work at the company did some task in a certain way that was lost when he was gone, and then a few months later his substitute had the same need and went ahead and created the same field with a slightly different name.
Doing data engineering work with Salesforce data is an exercise on archeology, psychology and organizational politics.
Slop is basically the ontological and teleological philosophy behind Salesforce very existence. Despite the official discourse that the "No Software" meant no infrastructure, no toil with updates and configuration, the subtext as intended for executives was very clear: "No need for you to be blocked by those pricks from engineering and their stupid, bureaucratic and gatekeeping rules".
"No software" was a call-to-arms to a certain subset of managers that were radicalized by Nicholas Carr's 2023 HBR article "IT Doesn't matter". It doesn't matter that Carr was a journalist and a writer with a masters in English that has never ever run even a small bodega, or has never managed an IT department. Anti-intellectualism and the abundance of capital brought in by the petrodollar that allowed the US government to run deficits year by year while exporting the ensuing inflationary effects to rest of world, would ensure that this message would ressonate and then even be amplified during the years of ZIRP and the Baillouts. Play fast and loose, first come, first served, a rising tide rises all boats and all that jazz. Wall Street favors bold, and the heck with the long term! This quarter will only live once!
Frankly, this is just poetic justice: Kill by slop, be killed by slop.
Typo: Carr wrote that in 2003, not 2023 for those of you who missed the foolishness and insane wreckage that article caused.
The lost business value and competitiveness caused by outsourcing IT overseas to unmotivated parties under Carr's premise is hard to put your finger on but I have seen the aftermath and it's pretty massive.
Something tells me Troy the Salesforce Admin/BD Analyst did not cause the SAAS infrastructure to go down.
And I think you're confusing crud with slop.
I've never worked somewhere that had a Salesforce integration which wasn't an eternal disaster. Have you?
Why is every company's Salesforce team absolute bottom of the barrel developers with super high churn, no responsibility, and little competency?
Something about the product and its positioning attracts catastrophe. That's what GP is talking about.
"Trust just got personal!"