This sounds big enough to require a black start. Unfortunately, those are slow and difficult.
If an entire nation trips offline then every generator station disconnects itself from the grid and the grid itself snaps apart into islands. To bring it back you have to disconnect consumer loads and then re-energize a small set of plants that have dedicated black start capability. Thermal plants require energy to start up and renewables require external sources of inertia for frequency stabilization, so this usually requires turning on a small diesel generator that creates enough power to bootstrap a bigger generator and so on up until there's enough electricity to start the plant itself. With that back online the power from it can be used to re-energize other plants that lack black start capability in a chain until you have a series of isolated islands. Those islands then have to be synchronized and reconnected, whilst simultaneously bringing load online in large blocks.
The whole thing is planned for, but you can't really rehearse for it. During a black start the grid is highly unstable. If something goes wrong then it can trip out again during the restart, sending you back to the beginning. It's especially likely if the original blackout caused undetected equipment damage, or if it was caused by such damage.
In the UK contingency planning assumes a black start could take up to 72 hours, although if things go well it would be faster. It's one reason it's a good idea to always have some cash at home.
In another life I worked as an engineer commissioning oil rigs and I’ve seen how tricky even a small-scale black start can be. On a rig, we simulate total power loss and have to hand-crank a tiny air compressor just to start a small emergency generator, which then powers the compressors needed to fire up the big ~7MW main generators. It's a delicate chain reaction — and that's just for one isolated platform.
A full grid black start is orders of magnitude more complex. You’re not just reviving one machine — you’re trying to bring back entire islands of infrastructure, synchronize them perfectly, and pray nothing trips out along the way. Watching a rig wake up is impressive. Restarting a whole country’s grid is heroic.
I remember talking to my ex's dad about his job, which involved planning refuels of a large nuclear-powered generation station in the Lower Midwest.
The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. It's a system basically built to produce "common accidents". It's amazing that it doesn't on a regular basis.
> The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true.
Funny thing is, those are the exact words I use when talking to people about networking. And realistically anytime I dig deep into the underlying details of any big enough system I walk away with that impression. At scale, I think any system is less “controlled and planned precision” and more “harnessed chaos with a lot of resiliency to the unpredictability of that chaos”
This is one of the key insights in my early SRE career that changed how I viewed software engineering at scale.
Components aren’t reliable. The whole thing might be duct tape and popsicle sticks. But the trick for SRE work is to create stability from unreliable components by isolating and routing around failures.
It’s part of what made chaos engineering so effective. From randomly slowing down disk/network speed to unplugging server racks to making entire datacenters go dark - you intentionally introduce all sorts of crazy failure modes to intentionally break things and make sure the system remains metastable.
Everything is chaos, seek not to control it or you will lose your mind.
Seek only to understand it well enough to harness the chaos for more subtle useful purpose, for from chaos comes all the beauty and life in the universe.
We would instead have HaaS, with monthly subscriptions for a license to use the house. Which can be randomly revoked at any moment if the company doesn't feel like supporting it is profitable enough, or if an AI thinks your electricity usage is suspicious and permabans you from using a home in the entire town.
A bit of a tangent, but I don't think this is it. There are plenty of species with plenty of shared norms, expectations, and trust - but no civilization. And, vice versa, many of the greatest societies have been riddled with completely incompatible worldviews yet created amazing civilizations. Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart!
The reason people work together is fundamentally the same reason you go to work - self interest. You're rarely there because you genuinely believe in the mission or product - mostly you just want to get paid and then go do your own thing. And that's basically the gears of society in a nutshell. But you need the intelligence to understand the bigger picture of things.
For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps. They even wage war over them and in efforts to expand them or control their territory. This [1] war was something that revolutionized our understanding of primates behaviors, which had been excessively idealized beforehand. But they lack the intelligence to understand how to bring their little societies up in scale.
They understand full well how to kill the other tribe and "integrate" their females, but they never think to e.g. enslave the males, let alone higher order forms of expansion with vassalage, negotiated treaties, and so on. All of which over time trend towards where we are today, where it turns out giving somebody a little slice of your pie and letting him otherwise roam free is way more effective than just trying to dominate him.
> There are plenty of species with plenty of shared norms, expectations, and trust
Citation needed on that one.
> Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart!
They spoke the same language, shared the same literature, practiced the same religion, had a long history of diplomatic ties. When the Persians razed Athens, they took refuge with the Spartans.
> For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps.
Again, I don't think this claim stands to evidence. The so called chimp war you mention is about a group of about a dozen and a huge fight that broke out among them. That doesn't support the idea that they are capable of 200-strong 'intricate' groupings.
"They spoke the same language" ... not exactly, the Spartans spoke Doric, while the Athenians Attic. (Interestingly, there is a few Doric speakers left [0].) While those languages were related, their mutual intelligibility was limited. Instead of "Greek" as a single language, you need to treat it as a family of languages, like "Slavic".
"shared the same literature" ... famously, the Spartans weren't much into culture and art, and they left barely any written records of their own. Even the contemporaries commented on just how boring Sparta was in all regards.
If we delve deeper into ideas about how a good citizen looked like, or how law worked, the differences between Sparta and Athens are significant, if not outright massive.
While those two cities weren't entirely alien to each other, had some ties, same gods, and occassionally fought on the same side in a big war, there was indeed a huge political and cultural distance between them. I would compare it to Poland vs. Russia.
Not "entirely alien, had some ties" is not it. They were part of the same cultural cluster, participated in the same games, traveled to the same sanctuaries, had mutual proxenies. The very fact that we know the opinions of several Athenians about Spartans is telling. We don't know what they thought of inhabitants of Celtic population centers, or Assyrian cities, or Egyptian ones. But we know what they thought of individual Spartans that they mention by name, biographical detail and genealogy.
I stand by my comparison to the Slavic nations of today.
Yeah, we have a lot of opinions of one another, yes we understand basic vocabulary of our cousins, though details in fine speech are another matter, yes, we are technically Christian, but still the political and societal difference between, say, Czechs and Russians is quite big.
As was the difference between the Spartans and the Athenians. Constitutionally, the poleis were all over the map, from outright tyrannies, through oligarchies and theocracies, to somewhat democratic states.
So your argument is: Athens and Sparta had things in common but were different. Like Czechia and Russia. Czechia and Russia are quite different. So were Athens and Sparta?
Try to speak holistically. I have no idea what you're trying to argue. I could expand or provide evidence for everything I said, but providing a citation or proving that there are indeed social groups of upwards of 200 chimps, or whatever, isn't going to do much, because you're not really formulating any argument or contrary view yourself.
Put another way, you're arguing against an example and not a fundamental premise. Proving the example is correct doesn't really get us anywhere since presumably you disagree with the fundamental premise.
That sounds very much like "Just believe me." or even more "The rules were that you guys weren’t gonna fact-check"
> I have no idea what you're trying to argue.
Presumably you know what you are trying to argue. That is what the questions were about.
> Proving the example is correct doesn't really get us anywhere
You would have solid foundations to build your premise from. That is what it would get us.
First we check the bricks (the individual facts), then we check if they were correctly built into a wall (do the arguments add up? are the conclusions supported by the reasoning and the facts?). And then we marvel at the beautiful edifice you have built from it (the premise). Going the other way around is ass-backwards.
> you're not really formulating any argument or contrary view yourself.
I don't know what viewpoint namaria has. I know that "Sparta and Athens [..] couldn't possibly have been further apart" is ahistorical. They were very similar in many regards. If you think they were that different you have watched too many modern retellings, instead of reading actual history books. That's my contrary view.
> For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps.
Here the question is what do we believe to be "societies". The researchers indeed documented hundreds of chimps visiting the same human made feeding station. Is that a society now? I don't think so, but maybe you think otherwise. What makes the Chimps' behaviour a society as opposed to just a bunch of chimps at the same place?
The preppers can only buy themselves a small amount of time, though—no more than a year or two. Eventually, their stockpiled supplies will run out, or some piece of equipment will need a replacement part.
I'd much rather focus on "prepping" by building social resiliency, instead. The local community I'm plugged into is much stronger together than anything I could possibly build individually.
I am an ex-scientist and an engineer and had a look at the books of my son who studies finance in the best finance school in the world (I am saying this to highlight that he will be one of the perpetrators, possibly with influence, of this mess)
The things in there are crazy. There are whole blocks that are obvious but made to sound complicated. I spent some time on a graph just to realize that they ultimately talk about solving a set of two linear equations (midfle school level).
Some pieces were not comprehensible because they did not make sense.
And then bam! A random differential equation and explanation as it was the answer to the universe. With an incorrect interpretation.
And then there are statistics that would make "sociology science" blush. Yes, they are so bad that even the, ahem, experts who do stats in sociology would be ashamed (no hate for sociology, everyone needs to eat, it is just that I was several times reviewer of thesises there and I have trauma afterwards).
The fact that finance works is because we have some kind of magical "local minimum of finance energy" from which the Trumps of this world somehow did not maybe to break from (fingers crossed) by disrupting the world too much.
I did a lot of work for a major airline earlier in my career and came away with same impression. I just couldn’t see how they kept planes in the air based on my experiences through out the organization. I think in a big enough org the sheer momentum keeps things moving despite all the fires happening constantly.
"Funny thing is, those are the exact words I use when talking to people about networking"
Computer networking is not the same. Our networks will not explode. I will grant you that they can be shite if not designed properly but they end up running slowly or not at all, but it will not combust nor explode.
If you get the basics right for ethernet then it works rather well as a massive network. You could describe it as an internetwork.
Basically, keep your layer 1 to around 200 odd maximum devices per VLAN - that works fine for IPv4. You might have to tune MAC tables for IPv6 for obvious reasons.
Your fancier switches will have some funky memory for tables of one address to other address translation eg MAC to IP n VLAN and that. That memory will be shared with other databases too, perhaps iSCSI, so you have to decide how to manage that lot.
You tried to nerdsnipe someone without mentioning L2 is effectively dead within datacenters since VXLAN became hardware accelerated in both Broadcom and "NVIDIA"(Mellanox) gear. And for those that don't need/care about L2 they don't even bother and run L3 all the way.
EVPN uses BGP to advertise MAC addresses in VXLAN networks which solves looping without magic packets, scales better and is easier to introspect.
And we didn't even get into the provider side which has been using MPLS for decades.
A problem with high bandwidth networking over fiber is that since light refracts within the fiber some light will take a longer path than other, if the widow is too short and you have too much scattering you will drop packets.
So hopefully someone doesn't bend your 100G fiber too much, if that isn't finicky idk what is, DAC cables with twinax solve it short-range for cheaper however.
I built control computers for nuclear reactors. Those machines are not connected to a network and are guarded by multiple stages of men with automatic machine guns. It was designed to flawlessly run 3x boards each with triple-modular-redundant processors in FPGA fabric all nine processors instruction-synced with ECC down to the Registers (including cycling the three areas of programmable fabric on the FPGAs). They cycle and test each board every month.
Well, the news says that doge randos are potentially exfiltrating the details of systems like that as well as financial details of many Americans, including those who hold machine guns and probably suffer from substandard pay and bad economic prospects/job security as much as anyone else does.
Perhaps the safest assumption is that system reliability ultimately depends on quite a lot of factors that are not purely about careful engineering.
A bit off topic, but my uncle used to be security at a nuclear plant. Each year the Delta Force (his words) would conduct a surprise pentest. He said that although they were always tipped off, they never stopped them.
I guess the biggest security advantage of any of these old critical systems is fact that they are not connected to the internet. At least I hope they are not.
The regulations around parts sourcing, required maintenance, and training has more to do with how well/safe modern aviation is than anything else. If those aren’t done properly, all sorts of weird things start happening. Pretty much the only reason aerospace safety records aren’t worse in third world countries is because of how obviously bad the consequences are quickly - and even then….
I love the "analog" handcranked air compressor to 7MW generator escalation, it really captures human ingenuity.
I wonder however how being part of the "continental Europe synchronous grid" affects this, and how it isolates to Portugal and Spain like this.
But yeah there are a lot of capacitors that want juice on startup that happily kills any attempt to restore power. My father had "a lot" of PA speakers at home and when we tripped the 3680w breaker (16A 220v) we had to kill some gear to get it back up again. I'm also very sure we had 230v because I lived close to the company I worked for and we ran small scale DC operations so I could monitor input voltage and frequency on SNMP so through work I had "perfect amateur" monitoring of our local grid. Just for fun I got notifications if the frequency dropped more than .1 and it happened, but rarely. Hardly ever above though since that's calibrated over time like Google handle NTP leap seconds.
I saw some ancient footage of an Me-109 fighter engine being started. A tech jumped on the wing and inserted a hand crank into a slot on the side. He threw all his might into turning it, and then after a delay the propeller started turning and coughed into life.
I realized the tech must have been winding up a flywheel, and then the pilot engaged a clutch to dump the flywheel's inertia into the engine.
The engineer in me loves the simplicity and low tech approach - a ground cart isn't needed nor is a battery charger (and batteries don't work in the cold). Perfect for a battlefield airplane.
---
I saw an exhibit of an Me-262 jet fighter engine. Looking closely at the nacelle, which was cut away a bit, I noticed it enclosed a tiny piston engine. I inferred that engine was used to start the jet engine turning. It even had a pull-start handle on it! Again, no ground cart needed.
---
I was reading about the MiG-15. American fighters used a pump to supply pressurized oxygen to the pilot. The MiG-15 just used a pressurized tank of air. It provided only for a limited time at altitude, but since the MiG-15 drank fuel like a drunkard, that was enough time anyway. Of course, if the ground crew forgot to pressurize it, the pilot was in trouble.
You are correct, the official moniker is Bf-109, but the Allies referred to it as the Me-109.
BTW, since we are Birds of a Feather, I bet you'd like the movie "The Blue Max". It's really hard to find on bluray, but worth it! The flying sequences are first rate, and no cgi.
Blackstart assumes *no* power is available, period. Nothing but human muscle power. Thus the first stage is always either a human pulling a starter cord or the like, or a human building up energy in some fashion that is then dumped into the system to produce a bigger surge than is possible by direct muscle power.
And, despite the news reports, this is not a true blackstart. Some power survived.
> have to hand-crank a tiny air compressor just to start a small emergency generator
Similarly, the US Navy maintains banks of pressurized air flasks to air-start emergency diesels. Total Capacity being some multiple of the required single-start capacity
Random fact: Those starters are a plot point in the 1965 film The Flight of the Phoenix, where the protagonists are trying to start a plane that’s stranded in the Sahara, but only have a small supply of starter cartridges left.
I lived for a while on a sailboat equipped with an ancient Saab tractor engine (8 whole horsepower!). Was designed for cartridge starts in cold weather, though someone had fitted an electric starter by the time I saw it
That would be charging up the spring to throw the breaker. High voltage breakers need to switch on (or off) very quickly, to avoid damage from arcing. It's common for them to have some kind of spring or gas piston arrangement that you pump up first to give them enough energy to do that quickly.
No, he's winding up a spring to close the circuit breaker quicker than a human hand could, which reduces/prevents and arc from forming as the electrical contacts close.
If you were the right age when it came out in theaters in '93 (roughly between 11-15), Jurassic Park was a huge deal. Titanic was another of those in that era (although mainly to certain females).
I can appreciate the ability to revert to hand cranking an air compressor, yet I can't help but feel that the 99.99% of events, you'd be better served with keeping a two stroke gas engine ready to go. Air compressors tend to have parts just as or more vulnerable to environmental factors, and you get a lot more power for less elbow grease out of a two stroke.
In 99.99% of real-world scenarios, the rig would have other options to bootstrap a black start—like fully charged air tanks, backup power from a support vessel, or even emergency battery systems. The hand-cranked air compressor is really a last resort tool. We test it during commissioning to prove it could work, but in most cases, it’s never used again in the rig’s working life. It’s there for the rarest situations—like if a rig was abandoned during a hurricane, drifted off station, and someone somehow ended up back onboard without normal support. It’s a true "everything else failed" kind of backup.
Nice to see that at least in some places people are actually thinking to almost-impossible scenarios and taking them into account. I have the feeling that it's quite unlike most infrastructure development nowadays, unfortunately.
The key is the responsible party's skin in that particular game. A drilling rig is a very large, very expensive, and very lucrative man-made island. The backed-up backups have backups. Not only could it be very far away from support vessels, capable of bringing it online in every situation, every minute not in production is money thrown overboard.
Very true, although I think that economic arguments can apply to most infrastructure. What are the actual costs of a day-long nationwide blackout? I have no actual idea, but I'd not be surprised if they exceed 1 billion {EUR|USD}.
The part you are missing is ‘paid by whom’. Unlikely the power companies or regulator is going to be paying that amount here. It’s all the poor saps who didn’t have sufficient backup capacity.
There will be costs/losses by the various power companies which weren’t generating during all this of course, but also fixing this is by definition outside of their control (the grid operators are the ones responsible).
I’m sure public backlash will cause some changes of course. But the same situation in Texas didn’t result in the meaningful changes one would expect.
That’s because there is no effective regulation of the state’s power industry. Since they’re (mostly) isolated from the national grid, they aren’t required to listen to FERC, who told them repeatedly that they should winterize their power plants. And a state-level, the regulators are all chosen by the Governor, who receives huge contributions from the energy industry, so he’s in no rush to force them to pay for improvements.
The real irony was the following summer during a heatwave, when they also experienced blackouts. Texas energy: not designed for extreme cold, not designed for extreme heat. Genius!
Same thing happened in south Texas last year. Years of deferred maintenance on transmission lines resulted in almost two weeks of power outages from two major storms, that could have largely been avoided. The utility provider is mostly allowed to regulate itself (while donating to the campaigns of the dominant political party), and allowed to keep excess profits/return dividends to shareholders, rather than re-invest in infrastructure. There is very little regulatory structure or checks in place to ensure the grid is being maintained. And there have essentially been no consequences, other than an apology and excuses, with an attempt to raise delivery rates even higher. As a home owner, its on me to bear the additional cost of a backup generator, because I can’t rely on the state to regulate the utility to provide the service I’m forced to pay them for.
Based on how difficult it can be to start my chain-saw, snow-blower, and motorcycle after they've sat without being run for a while, I'd not recommend a gasoline-powered engine to be the only thing on stand-by.
Air compressors in adverse environments don't hold up that well either, without basic maintenance. I've had engines run seasonally for decades. It doesn't take much for them to keep working well, though doing nothing at all is an easy way to clog up the carburator.
Compressor pistons/screws that ingest grit/dirt, or aren’t run often enough to boil the water out of them, also tend to not last long. I used to help run a volunteer workshop with an Atlas Copco screw compressor, and it died in a few years because it wasn’t being run hard enough and the screws rusted (doh!).
It shouldn't be that bad. A little fogging oil when put away and drain all the fuel. Then a little starting fluid on the first couple start attempts. Usually they start fairly quickly if they're in decent shape. And that's just for pull starts. My electric start mower starts right up after even 5 months of not running with stabilizer in the fuel.
As an ex small engine mechanic, I'd advise against using a 2 stroke for something like that. A 4 stroke would be a better bet. Better yet would be a natural gas/propane 4 stroke, since gasoline goes stale and plugs carburetors.
Small diesels could be an option but they're harder to pull start for a given size.
> Small diesels could be an option but they're harder to pull start for a given size.
I once needed to jump-start a small marine diesel, many miles from land...
There was a small lever that cuts compression. You have to get it spinning really fast before restoring compression! It's definitely a lot of work!
EDIT - Here is a cheap modern small marine diesel [1]. The operation manual suggests that you don't have to do anything to get it spinning quickly, you just have to crank it 10 times, put away the crank handle, and then flip the compression switch. That's progress!
Lister diesel generators are much the same - half a dozen cranks, restore compression and off they go. The hand cranking can easily break your arm if you get it wrong though.
Even gas engine pull starts have a compression release function built in. That's why you need special cylinder pressure tools to check compression on most pull starts.
I did that too and crank got stuck on flywheel. To stop engine I had to climb over the engine where now-removed stairs were since my mate was clueless. Fortunately the crank handle stayed on.
Cranks and decompression levers are gone for at least 30-40 years now tho.
Not being at all qualified to comment (though I work for a power company), I'd think the hand crank air compressor wouldn't suffer from no spark or bad gas.
If stale gas is a concern, then all of the other steps in-between zero power and your full start are also screwed.
Air compressors have more valves and gaskets that are vulnerable to oxidation, especially in salty environments, so I'd have thought the upkeep between the two, the two stroke would be easier.
But it's an emergency system, not a general operation system. Thus it's not going to be exposed to the salty environment most of the time. You could certainly put the whole thing in an airtight box.
Look at how the military builds surface-based missiles these days: it's in a factory-sealed box. Molten salt batteries so they last for decades. (You don't see molten salt in most purposes because once it's been triggered it's lifespan is in minutes or even less. They're used in applications that only need to deliver power once.)
Diesel will run on mostly anything if it’s running… including methane in the air intake, so you need to think quickly when presented with a generator that keeps running after cutting the fuel
Oil leaking around a turbocharger rotor seal also makes for good diesel fuel, if you define "good" as an exciting uncontrolled disassembly of the engine.
Crude oil from various wells has properties varying from ‘thick, stinky, corrosive goo’ to ‘explosive, barely liquid, bubbly mess’. Also, rigs need to be careful about ignition sources, as methane leaks can be a common emergency condition for some wells/crude.
It’s not the type of thing that using directly is economically feasible, even for emergency situations.
Batteries are great when they have charge. What happens if the generator doesn't want to start the first, second, and third time? How many start attempts do you get before the batteries are dead?
The hand-pumped air compressor is the tool of last resort. You can try an engine start if there's someone there who's able to pump it. You don't have to worry about how much charge is left in your batteries or whether or not the gasoline for the 2-stroke pump engine has gone stale. It's the tool that you use as an alternative to "well, the batteries are dead too, guess we're not going to start the engine tonight... let's call the helicopters and abandon ship"
The data center where I work has large diesel generators for power cuts. They are electric (battery) start. There is no capability to start them manually. The batteries are on maintenance chargers that keep them in good condition. The generators are started and tested every two weeks.
Could the batteries be dead and the generators not start? I guess but it's very unlikely. I get that on an oil rig it might be a matter of life and death and you need some kind of manual way to bootstrap but there's not much that's more reliable than a 12V lead-acid battery and a diesel engine in good condition.
Also, the data center is probably in a city, surrounded by infrastructure that could be used if necessary. An oil rig is in the middle of an ocean, and has to rely on itself.
Lead acid batteries are not exactly what I would call reliable. They require a lot of constant maintenance to ensure that they will work when you need them and they can easily degrade in such a way that they maintain voltage and appear to be good but then fail to deliver the needed amps when you demand them. This is made much worse in cold weather. Finally, if allowed to freeze when they are moderately drained, then the accumulated water inside will freeze and drastically shorten their life span.
I think I'd take Lithium Ion batteries over lead acid for almost every conceivable use-case. They are superior in almost every way. Lighter, less likely to leak acid everywhere, better long term storage (due to a low self-discharge) and better cold weather discharge performance. The only drawback would be a slightly increased risk of fire with Lithium.
I worked with a telecom provider's data center that ended up having a quad redundant diesel generator failure during the first cold snap that took the Texas grid offline a few years back. They had at three fuel supplies gel and then failed to start. The fourth, as I remember, just didn't try to fire.
It's unrealistic, and if one power station is unable to use their batteries to start their emergency generator (through the absurd incompetence you describe, or more likely through a major fire, flood or assault) the grid can be started from a different one.
Black out on a rig or ship is very different to black start of a national electricity grid.
Most vessels will experience a blackout periodically and the emergency generator start fine, normally on electric or stored air start, and then the main generators will come up fine. It's really not delicate, complex or tricky - some vessels have black outs happen very often, and those that don't will test it periodically. There will also be a procedure to do it manually should automation fail.
There are air starters on some emergency generators that need handling pumping. These will also get tested periodically.
The most complex situation during black out restoration would be manual synchronisation of generators but this is nothing compared to a black start.
The point isn't to make a system that is easy. The point is to make a system that is guaranteed to work in any remotely realistic circumstance.
In a real black start, the guys might very well grab a portable generator and just use that instead. But having the option to hand crank something rather than rely on batteries that might run flat is good.
and if the entire thing depends on it, you'll give that generator a handcrank as a backup too instead of assuming the batteries ever dying or getting flooded or whatever is entirely impossible.
bringing islands together requires one to synchronize both -- frequency and phase. It is super difficult for large generators and transmission lines. transient heat dissipation can be a real bummer.
How hard is getting each island within .1Hz of correct? The full grid doesn't have much trouble, but I don't know how much cutting things down impacts that.
And then phase will align itself a couple times a minute so what's difficult about that part?
The compressor pressurizes an air tank. When the pressure in the tank is nice and high, use the compressed air to turn a turbine connected to the crankshaft of the engine.
You can also directly feed the compressed air into a cylinder (or even the intake manifold!) to force the engine to turn. No extra turbine required, though the plumbing might get a little odd. [https://en.m.wikipedia.org/wiki/Air-start_system]
That tends to be for very large engines, where the extra plumbing isn’t a problem.
This technology of starting a diesel engine using a turbine driven by compressed air was used in Russian T-34 tank during the WWII. While Germans could not start the tanks in the cold of winter 1941 from the frozen batteries the Russians were using compressed air (hand-crank) to start T-34s just fine.
The fewer resources we dedicate to grid resilience and modernization, the harder black starts become. And as grids get more complex and interdependent, recovering from total failure becomes exponentially harder.
A rare but sobering opportunity to reflect on something we usually take for granted: electricity.
We live in societies where everything depends on the grid — from logistics and healthcare to communications and financial systems. And yet, public awareness of the infrastructure behind it is shockingly low. We tend to notice the power grid only when it breaks.
We’ve neglected it for decades. In many regions, burying power lines is dismissed as “too expensive.” But compare that cost to the consequences of grid collapse in extreme weather, cyberattacks, or even solar storms — the stakes are existential. High-impact, low-frequency events are easy to ignore until they’re not.
Just to highlight this: the last significant power outage in Western/middle Europe was 2003. [1]
That's 20 years without any significant problems in the grid, apart from small localized outages.
It's not hard to start taking things for granted if it works perfectly for 20 years.
Many people don't even have cash anymore, either in their wallet or at home. In case of a longer power outage a significant part of the population might not even be able to buy food for days.
> Many people don't even have cash anymore, either in their wallet or at home.
Even if you have cash many shops would not sell anything in case of a mass outage because registers are just clients which depend on a cloud to register a transaction. Not reliable but cheap when it works.
Many supermarket chains (in the West at least) have satellite links at their major locations because they can't afford to close a store just because the local ISP had an issue.
The real question is how long can some of the smaller banks' datacenters stay up.
Firstly and most importantly, a cash register needs a power outlet. It is highly contestable that every single Western supermarket out there has a diesel generator down in the back / storage room that will kick in in an instant if a power outage begins.
Lest also forget the Crowdstrike drama where many supermarkets simply went dark, in some instances for nearly 24 hours, despite working communication links. But I digress.
Crowdstrike was an interesting one. Just as it was going down I went out to the supermarket and found half of the self checkouts had bluescreened. Then a few hours later they were all back and functioning again. The supermarket had remote management at a level below the OS that could restore the whole countries self checkouts rapidly.
I would not be surprised if they simply booted an image from the network. It would significantly simplify maintenance, as for any change you'd just need to update a single image and push it downstream to an in-store management server. The individual terminals essentially become disposable.
> It is highly contestable that every single Western supermarket out there has a diesel generator down in the back / storage room that will kick in in an instant if a power outage begins.
Literally true. However:
- If it takes them 10 minutes to fire up the generator, then 5 minutes to restart the network and registers, that is no big issue (in a many-hour outage)
- At least in my part of the USA, many supermarkets do have generators - because storm damage causes local outages relatively often, and they'd lose a lot of money if they couldn't keep their freezers and refrigerators powered. Since the power requirements of the lighting and registers are just (compared to the cooling equipment) a rounding error, those are also on generators.
Plus, there are backup-power lories and refrigerated trailers. If your shop doesn't have enough backup power for duration, you might see several of these pull into the carpark all at once. If not all of the chillers can be powered, shop's staff will schlep stuff to the refrigerated lories.
Seen it done in USA, for a Target next to a Kroger grocer. Kroger lost everything that needed cold after reserve either ran out or wouldn't start, but Target had a contingencies contract and lost no product.
Well,in my experience it was the case for the 2 largest supermarket chains. We lost electricity at 12:30pm and only got it back during the night at 3am.
But both major supermarkets nearby worked on diesel generators and payment by card worked flawlessly. I guess they had satellite connection.
It might have been more complicated in small villages but people living in rural areas ually still use a lot of cash.
In my local area of Spain/portugal, 2/3 supermarkets and 2/3 gas stations had generators up and running within a couple of hours. We’re pretty rural, though - I don’t know that urban areas faired as well
We're outside Mataro, had to make a trip into Barcelona yesterday. I'd say most gas stations north of Barcelona/Maresme area were 100% offline, some (we found only two, from 6 visited) gas stations still had operational pumps but huge queues and cash only. None of the TPVs seemed to work anywhere in the afternoon yesterday here, even the battery powered/mobile network ones.
That's true, I went shopping cca. 4-5 hours after the blackout started and had no issues, even card transactions worked. Whoever designed the retailer, they clearly had this scenario in mind. Even the "self-service" computer kiosks all worked.
In a multi-day event like we are talking about here, couldn't a shop owner revert to a paper ledger? I mean, it would suck and transactions would take much longer but if the alternative is people starving or having your inventory looted by a desperate mob, a nineteenth century solution seems preferable.
They can and do. They will also do deep cuts on prices/give away for free for refrigerated and frozen goods because those will just get tossed and neighborhood goodwill is still a thing in some places.
There was a 4-day power outage here (Seattle suburbs) last fall, and one of the auto parts store made an effort to serve customers even though they didn't have any power. I paid cash, and I forget whether or how they did credit card transactions (possibly by writing down CC numbers on paper.) They made a lot of phone calls to a different store to get prices for items.
I'm located in Barcelona, and yesterday lot of transactions on mini markets / pharmacies were not possible because the item prices were unknown, adding to the fact there was no phone lines available to reach out.
Portugal has mandatory electronic receipts. By this I don't mean email receipts, I mean that all receipts have a code on them that is then also available to be looked up on the government's side (e-Fatura is the app for this) for tax reasons. I think it's fairly simple though, just registers the total amount, the seller, and how much VAT was paid.
However I assume this can work offline with the data being uploaded later though, as basically all the small supermarkets and shops were still open here (_incredibly_ chaotic though), and on the big supermarkets card payments were working (TBF, even the free wifi was working there, I guess they probably have some satellite connection).
> Many people don't even have cash anymore, either in their wallet or at home. In case of a longer power outage a significant part of the population might not even be able to buy food for days.
So, what's really interesting is that these sorts of social collapses have happened. In fact, they often happen when natural disasters strike.
When they do happen, mutual aid networks just sort of naturally spring up and capitalism ends up taking a backseat. All the sudden worrying about the profits of Walmart are far less important than making sure those around you don't starve.
As it turns out, most people, even managers of stores, aren't so heartless as to let huge portions of the population starve. Everyone expects "mad max" but that scenario simply hasn't played out in any natural disaster. In fact, it mostly only ends up being like that when central authority arrives and starts to try and bring "order" back.
You can read about this behavior in "A Paradise Built in Hell" [1].
Well if the situation happens that people can't buy food, things will easily become nasty quickly. People will break open stores and use violence to get what they need. So cash money won't really help a lot in a serious situation.
The "civism" was well noted in the Spanish case, as far as I know, the whole country passed this incident without a single case of security issue. During the whole thing, people were extremely kind and polite. People sitting in dark bars was kind of "funny" how people clinging to their normal lives even in emergencies.
I often wonder if we should leave energy/telecommunications in a state where they can and do fail with some degree of frequency that reminds us to have a back up plan that works.
I had thought that the (relatively) recent lockdowns had taught us how fragile our systems are, and that people need a local cache of shelf stable foods, currency, and community (who else discovered that they had neighbours during that time!)
For something like this, a local electricity generation system (solar panels, wind/water turbines, or even a ICE generator) would go a long way to ensuring people continued to have electricity for important things (freezers)
There are now ubiquitous wireless POS terminals for card payments that can be recharged from emergency sources of electricity(like cars). As long as the mobile internet works it's possible. Of course this only little alleviates the disruption.
Surprisingly mobile networks seemed to stay up in Portugal. I'm not sure to what extent and if they lasted for the whole duration of the blackout.
They definitely limited consumer use though.
Most mobile base stations have a limited backup battery and some have generators on site. I'd expect telephone infrastructure to have 24-48 hours of backup in the USA and I don't know why Europe would be much different.
Population density is pretty high in many European inner cities. Most of the cell sites around here are on top of apartment buildings and I doubt they have a genset. Here in central BCN the mobile network was completely offline within an hour or two of the power going out.
Surely there are still card POS systems that can buffer transactions? Sure you loose some part of the system like payment authorizations but the potential loss of money is lower than closing shop.
The 1000EUR limit doesn't apply between private individuals; for businesses, you will find many other European countries also don't take large cash payments, for security or convenience reasons. E.g. you can't buy a car from a dealership for cash in "cash is king" Germany either. They expect a wire in almost all cases.
I don't know but I find it very practical to not carry piles of cash in my pocket and home and know that we're less likely to get robbed just because of the cash we have.
I don't know how true the relationship between the cashless lifestyle and safety actually is, but it works and I feel ok; I'm not sure that the prospect of a few hours of national blackout once in 20 years will make me change my mind significantly.
Today I was able to walk into a grocery store, pay for food, and go home to have a warm lunch (having a gas stove also helped tremendously). The matter was having a 10€ note at home. Not what I'd call "piles of cash".
As an added benefit, no bank knows where I bought and when, which I find is a great advantage over the alternatives. (I also use Gpay; this comes from someone who just found a good middle point without forgetting about the more reliable, physical and privacy friendly option)
I think I have 10€ laying around at all times, possibly in loose change between my home and car. I do not always walk around with that in my pocket and I never have more than 500€ at home.
I didn't mean literally zero cash, but once the bulk of your transactions are by card, you don't need to constantly go to the ATM and replenish your cash reserves
And I myself didn't mean that I happened to have a measly 10€ note at home, I normally have around 300 minimum, to spend organically on purchases.
Of course I get that carrying coins and notes is cumbersome, but if we've managed to live all through the 80's and 90's with it, I think we can manage to keep doing it. 100% digital money is giving up on a huge level of self-determination and privacy that I wouldn't feel comfortable with, but I guess as newer generations grow up already pre-adoctrinated and not being able to compare the before-and-after, in the end society will end giving up.
I got my first cell phone as a full bearded adult so I do remember the times where you carried cash and the time you could actually meet people in places without having to constantly update one's position.
I don't think it's just a generational divide.
I do understand the privacy and self-determination problems of a cashless society but I have to admit I'm just to weak-minded to care about that in practice; the practicality of just paying even for just coffee with my phone is just too big for me to care for it.
> the time you could actually meet people in places without having to constantly update one's position
Not sure I understand how that's different than today? You set a time and place, then you meet there, are people doing more than that today? Seems the youngsters understand this concept as well as older people, at least from the people I tend to meet like that.
> the practicality of just paying even for just coffee with my phone is just too big for me to care for it.
Interestingly enough, no matter if you had cash or card yesterday you couldn't get a coffee anywhere, as none of the coffee machines had power and even in the fancier places where they could have made the coffee without power, they didn't have electricity for the grinder itself, so no coffee even for them.
> You set a time and place, then you meet there, are people doing more than that today?
no, today people are continuously updating you about their whereabouts and assume you can just change time, place continuously and if you don't have the phone people get lost and panic. Ok I'm exaggerating of course, but there is a grain of truth in this
It does seem safer to not carry cash. However, I remember around May 2024 that there were some reported incidents in Chicago were in the early morning hours, there were some groups of robbers who would force victims to do something I found especially worrying: they would be forced into resetting their phone password and logging into their mobile banking app. I’m not sure what ended up happening in these cases.
I once used an aggregator app to summon a handyman to my place. My request was simple: move two pieces of furniture around my very small apartment.
So I find a reputable service within the app, I schedule it, and they send a guy. He shows up to my door breathless, with some kind of sob story about a vehicle breakdown. I dismiss that out of hand and he gets to work. He did a fine job and it didn't take very long.
Then we get to the point of settling up, so I announce I'm going to pay in the app. He looks really disappointed and says he usually takes cash. I realized at this point that he was ready to shake me down, and also he would incidentally be discovering where I stashed my cash, when I reached for it with him there in the room. So disappointed. So I send the money out in the app and I show the confirmation screen to the guy. And I felt so bad that I followed up with a tip in the same fashion.
But at the end of the day he was just a garden-variety cash-in-hand scammer and I had no reason to feel guilty, because I had unwittingly outwitted him by trusting the app. And the company had no qualms about it.
Another time, I had a very short cab ride to the laundry. And it did not take long for the driver to spin a gigantic tale about his auntie addicted to gambling used up all their savings and they was really hurting for money. I was shifting uncomfortably wondering why I was hearing this. So the cabbie parks the car and his POS machine shuts off. He's like "oh it's out of order" so here he is, shaking me down and expecting me to go fetch cash to put in his grubby hands.
I stared him up and down, started taking photos, and got out of there. I discussed with dispatch. They said if he's not accepting cards and I intended to pay by card, I owe him nothing.
So again a cash-in-hand sob-story scammer was foiled. The cab service was crazy enough to assign him to pick me up additional times. This is why I ride Waymo, folks!!!
This might be one of the most paranoid things I've ever read on HN.
The laborer was simply trying to actually get paid vs. deal with the overhead of the app. Somewhat shady perhaps - since it routes around the company taking their cut for finding him the work, and likely avoids taxes. I've paid these sorts of guys cash every single time I've used such a service and exactly zero of them have "shook me down" or cared where I stored the money. They make so little already I'm happy to help them out with a smile.
Cabbies simply want cash for pretty much the same reason. They get charged an astronomical "service fee" by the cab company, and likely are avoiding taxes as well. I agree that such a situation is more shady in general, but I've actually had (NYC) cops side with cabbies on this topic and force me to go get cash at the ATM or get arrested. I also use car services now over cabs whenever possible due to this reason - mostly for convenience, never out of fear of being robbed though.
The chances of you getting mugged/stolen from for using cash are just the same as the chances of you getting mugged for no apparent reason walking home. Perhaps the collective dis-use of cash has reduced these odds, but you specifically is utterly irrelevant.
Lol what? How do you know either of these guys were trying to rob you? They prefer cash because they can take it under the table and not report it for taxes. Everybody knows that.
Yes, well, I choose not to participate in shady shit like that. Is that OK that I prefer to make transactions as laid out by their employer and not every random guy?
> happy to help them out with a smile
So you choose to be knowingly complicit in tax-avoidance schemes. That's fine; you do you, but some of us steer clear of shady shit, just on principle, you know? Perhaps the company deserves their cut as well -- they get paid so little already, amirite?
Also if there was nothing unusual about their choice of payment, then why must they regale me with these shitty sob stories? Am I supposed to be moved to tears at their hardship and heroism at making it to my door, that I must promptly cover their expenses? They are not panhandlers, they are service providers.
No, I ordered a service and I pay for the value of the service, according to the Company's rates. The cab company was clear about it: either I pay how I want to or I don't owe them. Nobody's arresting me for refusing to fork over cash. That's a scam.
That's not the only time I was cash-scammed by a cab driver. They will pull every trick in the book, and surely they compare and trade notes on their marks.
It gets even worse: my simple insistence on transacting with the cab company earned me fake receipts. Yes, they faked every receipt that they sent me in email. The totals were all fudged down to be much smaller than what I paid, including a $0 tip. It was very very obvious, especially when the rides booked in the app were generating duplicates showing different calculations. I reported it twice to their backend developers and they said that there were some coding errors in device drivers; please stand by for a fix. LOL!
That's a scam to ensure that taxpayers can't get reimbursed for out-of-pocket medical expenses. Most/all cab companies provide NEMT services as well, and they can't stand when people go outside of insurance companies. So they falsified my receipts.
We will use better technology for electronic transactions. Most of banks worldwide still use COBOL for most (all?) their software infrastructure.
You can do as many electronic transactions as you wish without internet or electricity, provided you have something with charged battery. Problem is the transaction cannot be verified without internet, but when internet gets restored, all transactions can be applied.
That technology exists for more than a decade, so banks will implement it in 20 or 50 years. Most sane people will not wait patiently for half a century till some software engineer implements electronic transactions with COBOL, and we will use some kind of blockchain much sooner than that.
I see no problem with COBOL. I'd rather have my credit card transaction be processed with 40yo well tested COBOL than Java from the newest intern using copilot.
> Most of banks worldwide still use COBOL for most (all?) their software infrastructure.
Nahhh - some banks have some parts of the infrastructure in COBOL. Specifically larger retail banks often have their ledgers in COBOL. Most of them want rid and are actively getting rid. Most places have had programs to root COBOL out since before 2000, but there are residual implementations remain. The ledgers are the hardest place to deal with because of the business case as well as the awkwardness. Basically there's not much of an advantage (or at least hasn't been) in modernizing so keeping the thing going has been the option. Now people want to have more flexible core systems so that they can offer more products, although not so sure that customers want this or can consume it. Still - it supports the idea of modernisation so not many people are keen to challenge.
The most common big implementations I come across are in Java.
They use Go as well, I know someone who writes it for banks, but most of their infrastructure is written in COBOL. There are some sources [1], and some people have told me the same in person, not in exact numbers or percentages, but roughly the same.
Anyway point remains, electronic transactions with no internet or electricity is a solved problem, and banks don't want to solve it or they can't due to incompetency or maliciousness.
Currency transactions worth their weight in gold, it is of utmost importance for transactions to always be published to a central authority right away. If they don't have to be published, they should not exist at all. Imagine people buying stuff without anyone knowing right away! That should never, ever exist, for any reason.
It’s not incompetence it’s regulations and inertia. Getting your central banking system past the regulators in most countries is difficult enough that it shades the effort required to replace the core systems.
Yeah, this is the turkey’s dilemma - life on a farm is a lot better than life in the wild for 51 out of the 52 weeks of the year.
Most of our modern economy and systems are built to reduce redundancy and buffers - ever since the era of “just in time” manufacturing, we’ve done our best to strip out any “fat” from our systems to reduce costs. Consequently, any time we face anything but the most idealized conditions, the whole system collapses.
The problem is that, culturally, we’re extremely short-termist- normally I’d take this occasion to dunk on MBAs, and they deserve it, but broadly as a people we’re bad at recognizing just how far down the road you need to kick a can so you’re not the one who has to deal with it next time and we’ve gotten pretty lazy about actually doing the work required to build something durable.
"Just in time" is a phrase I hate with vehement passion.
You aren't optimizing the system, you're reducing safety marigns - and consequences are usually similar to Challanger.
This is a solution that teenager put in management position would think of(along with hire more people as solution to inefficient processes), not a paid professional.
Systems like electric grid, internal water management (anti-flood) shouldn't be lean, they should be antifragile.
What's even more annoying that we have solutions for a lot of those problems - in case of electric grids we have hydroelectric buffers, we have types of powerplants that are easier to shutdown and startup than coal, gas or wind/solar(which cannot be used for cold start at all).
The problem is that building any of this takes longer than one political term.
Things which can’t self improve can’t be antifragile by definition. NNT alludes to this multiple times - systems together with processes and people running them can be antifragile, but just things cannot.
I postulate the grid as a whole is antifragile, but not enough for the renewable era. We still don’t know what was the root cause of the Spanish blackout almost 24h after it happened.
JIT isn’t about reducing safety margin. It was pioneered by Japanese companies, namely Toyota. They are known for risk adverse, safety first.
> This is a solution that teenager put in management position would think of(along with hire more people as solution to inefficient processes), not a paid professional.
What kind of comment is this? Toyota has been using and refining it for decades. It wasn’t invented yesterday by some “teenagers”. Such a state of HN’s comment section.
JIT is definitely not perfect as exposed during the Covid period, but it isn’t without merits and its goal isn’t “reducing safety margin”.
Sure it is. That's exactly how it achieves the higher profitability. Safety margin costs money. Otherwise known as inefficiency.
Slack in the system is a good thing, not a bad thing. Operating at 95% capacity 24x7 is a horrible idea for society in general. It means you can't "burst mode" for a short period of time during a true emergency.
It's basically ignoring long tail risk to chase near-term profits. It's a whole lot of otherwise smart people optimizing for local maxima while ignoring the big picture. Certainly understandable given our economic and social systems, but still catastrophic in the end one day.
It literally is reducing safety margin(buffers) of a whole distribution system by definition, and it is also being applied in places where it does not fit - like systems that should be resilient to disruption and/or anti-fragile.
I would expect a paid professional in management discipline to be aware of such nuance but alas proven wrong again.
Challenger wasn't really about cutting safety margins, but about kicking the can on a known problem: blowby in the motor joints. It was a gut feeling by the engineers that the problem was related to temperature, but there was enough of a random element to it that there was nothing specific to point to.
That should have been enough to scrub anyway, but there was clearly political pressure to launch.
I do agree that they need to specifically design anti-fragile.
For the people who died of normally preventable death during covid while the health services were overwhelmed, the damage is irrecoverable. The chips shortage lasted years. Every year we become more, not less, dependent on the supply chain working. Every year we become less, not more, resilient.
I don't think it's crass to separate the deaths that occurred from a novel disease from the impact it had in society. In the medium term, it's a blip, never mind the long term. There's a huge chunk of society that thinks there was a huge overreaction!
The chips shortage has been difficult, but it's also been little more than an inconvenience when you look at it in terms of goods being available to consumers or whatever.
That chunk is heavily influenced by the propaganda that over a million dead people isn't a big deal. The propaganda is economically incentivized because slowing down the economy is bad for business even if it protects human lives.
Nearly all collapses are of limited temporal duration (except for extinction events....). I think it is fair to call a health system that failed to protect the nation and world a collapse. It failed to perform its function in a dramatic way. Now, the fact that most of us survived at least is being exploited to say it was no big deal and actually, why not trash every public health institution so the economy is never shut down again?
Sad whomp whomp horn: the economy is going to be negatively affected by covid disability and death on an ongoing basis and a new pandemic will still cause so much fear the economy will shut down.
We have plenty of small scale collapses that weren't of limited duration. It's just that such things are typically only noted by archeologists. We only see the survivors and thus conclude that collapse isn't an existential problem.
I do agree on Covid disability. Early on we saw some pretty dire predictions, but since then it's mostly been an exercise in muddying the waters. Lots of wheel-spinning about what constitutes long Covid when they should have simply been collecting data on the various symptoms. Better to not see the problem than have to deal with it.
Look at how we were handling AIDS before we discovered it was HIV destroying the immune system. Long Covid is still at that stage--we are seeing a slew of highly varied effects rather than the mechanism.
The post you are replying to is not talking about the Covid deaths, but rather about the deaths from other causes triggered by the Covid disruptions. When a trauma case dies of the lack of a ventilator because they're all in use on Covid patients. When the trauma case bleeds out at the scene because the ambulance is running a Covid patient to the hospital.
And a lot of people thinking it was an overreaction proves nothing. People don't get a vote on reality.
Honest question, are we better off in the long run, and is it a better solution, to decentralize energy generation and make more smaller grids rather than linking them all up? This isn't to say completely getting rid of the ability to transfer between the smaller grids to assist with power disruptions but to decouple and make it less likely for catastrophic "global" failures like this.
With a high fraction of renewables, the reverse is probably better in the long run. The larger geographic area you connect, the less you're affected by weather systems, and the wider area you can draw dependable dispatchable power such as hydro from. But that depends on having enough grid capacity to move enough power around, which is currently a problem.
But I wonder from a reliability (or lack of cascading failures) point of view whether synchronous islands interconnected with DC interconnects is more robust than a large synchronous network?
We're slowly reaching this point with the internet too.
I feel like to many technologists, the internet is still "the place you go to to play games and chat with friends", just like it was 20 years ago. Even if our brains know it isn't true, our hearts still feel that way.
I sometimes feel like the countries cutting off internet access during high school final exams have a point. If you know the internet will be off and on a few days a year, your systems will be designed accordingly, and if anything breaks, you'll notice quickly and during a low-stakes situation.
Maybe a good reason (in parts of the world where this is practical) to have some solar + battery storage. Doesn't even need to fully replace grid power, just enough to run the barebones when the grid goes out.
Houses have water tank that work as buffer for when the water stop, couldn't residential batteries work the same way?, it could detect drops in voltage and stop charging, so even if the grid is unstable in a black out, the residential isn't going to light up immediately, or only the houses without batteries
Interestingly it seems that the black start drill is considering a smaller zone of impact than what has happened here.
Also I suspect there is far more renewables on the grid now than in 2016.
This is potentially the first real black start of a grid with high renewable (solar/wind) penetration that I am aware of. Black starts with grids like this I imagine are much more technically challenging because you have generation coming on the grid (or not coming on) that you don't expect and you have to hope all the equipment is working correctly on "(semi)-distributed" generation assets which probably don't have the same level of technical oversight that a major gas/coal/nuclear/hydro plant does.
I put in another comment about the 2019 outage which was happened because a trip on a 400kV line caused a giant offshore wind farm to trip because its voltage regulator detected a problem it shouldn't have tripped the entire wind output over.
Eg: if you are doing a black start and then suddenly a bunch of smallish ~10MW solar farms start producing and feeding back in "automatically", you could then cause another trip because there isn't enough load for that. Same with rooftop solar.
This is potentially the first real black start of a grid with high renewable (solar/wind) penetration that I am aware of.
The South Australia System Black in 2016 would count - SA already had high wind and rooftop solar penetration back then. There's a detailed report here if you're interested:
"Grid tied solar won't put power into the grid when the grid is down. It's the one reason I didn't grid tie."
Why would that prevent you from being grid-tie? I have 53 panels (~21kw) grid tied and pushing to the grid, but in the event of grid failure my panels will still operate and push into my 42kwh battery array which will power the entire house. ( The batteries take over as the 'virtual grid source). I can then augment the batteries with generator and run fully off grid for an extended amount of time ( weeks in my case ).
these are mutually exclusive. What you have is a hybrid system, which is something i explicitly did not mention. A grid-tie system is not generating an AC waveform when the grid is down, at all. it cannot, by definition, and by design, as the AC waveform requires the grid to synchronize to.
Youre missing my point. I know that; I mean if you are restarting the grid and say you have a segment you think has 5MW of load that will come online, but you connect it and a minute or two later suddenly 2MW of grid tied solar detects the grid and starts exporting and you now have 3MW of load it is going to make it much more tricky to balance the restart. I'm not sure how much of a problem this is in reality but it seems to me restarting a grid is made much more tricky when you have millions of generation assets with no control over.
This is why most of the restart of the grid is being done as the solar input tappers out I suspect. The grid was pretty much down during peak sun and started coming back online around 5:30 pm or later.
> I mean if you are restarting the grid and say you have a segment you think has 5MW of load that will come online, but you connect it and a minute or two later suddenly 2MW of grid tied solar detects the grid and starts exporting and you now have 3MW of load it is going to make it much more tricky to balance the restart.
I really thought that sentence was going to end with "it makes it a lot easier to handle that segment".
Yeah you have some big problems if it's a complete surprise, but your status quo monitoring would have to be very strangely broken for it to be a complete surprise. Instead it should be a mild complicating factor while also being something that reduces your load a lot and lets you get things running quicker.
Practical Engineering did a really great video a few years ago on why black starts are hard, complete with a tabletop demo about the physics of synchronizing large spinning generators: https://www.youtube.com/watch?v=uOSnQM1Zu4w
Yes because they have to bring it all back up in phases so that they only face the load spike* from one interconnect at a time, which can take some time and can fail if there’s unknown damage like the GP said.
It really depends on the region though because almost all large hydroelectric dams are designed to be primary black-start sources to restore interconnects and get other power plants back up quickly in phase with the dam. i.e. in the US 40% of the country has them so it’s relatively easy to do. The hardest part is usually the messy human coordination bit because none of this stuff is automated (or possible even automatable).
* the load spike from everyone’s motors and compressors booting up at the same time
Presumably emergency phone comms still work(?) so they could issue instructions to do a phased (heh!) restart to avoid every fridge/air conditioner/whatever restarting at once. Not sure how successful that would be however.
There’s usually two parallel processes going on during a black-start: spinning the power plants back up to get them synchronized to the grid, and getting power back out to all the consumers. Power plants have breakers that disconnect them from consumers which they keep disconnected until independent connections to other power plants allow them to spin up their massive turbines and synchronize them to the grid. There’s also tons of other downstream breakers at substations which will be in various states of functionality.
The power plants with direct connections have hard lines and black-start procedures that get power out to the most important customers like telecom infrastructure, which provides the rest of the comms. In a real world full restart it’s going to mean organizing workers at many substations to babysit old infrastructure so cellular is pretty much mandatory.
And to add more fun, this time they're not dealing with a small number of individual power plants that can be connected with only some phone-call based coordination.
Instead, there are literally hundreds of smaller wind/solar installations. Some of which depend on rapidly fading cellular communication to restart. And some might need an actual site visit to throw on the physical breakers.
Spain is but Portugal is only connected to Spain and they are currently doing a full black start.
For Spain the external power and synchronization can come from France rather than generators which will help, but the process and complexities are still mostly the same. Call it a dark start, perhaps.
Sure. Portugal is in deed completely “isolated” if it loses Spain, so it needs black start capability if it wants some degree of autonomy.
As far as containing the issue, this was a disaster. On the flip side, this was as good an opportunity to test a black start as any, it went reasonably well, and the network operator was already in the process of contracting two further dams for the ability.
> A black start is the process of restoring an electric power station, a part of an electric grid or an industrial plant, to operation without relying on the external electric power transmission network to recover from a total or partial shutdown.[1]
The key part of the phrase is actually "electric power station, a part of an electric grid or an industrial plant." Note how the definition doesn't include an entire grid.
Only the first power plant in a black-start (like a hydroelectric dam or gas plant started by a backup generator) is truly "black started." The rest don't fit that definition because they depend on an external power source to spin up and synchronize frequency before burning fuel and supplying any energy to the grid. If they didn't, the second they'd turn on they'd experience catastrophic unscheduled disassembly of the (very big) turbines.
Only the first power plant can come online without the external transmission network.
You’re absolutely right, there are a lot of variations. In this case I think Portugal started from their own hydroelectric dam and restarted everything North to South while Spain started several in parallel from the interconnects to the rest of Europe (can’t tell which interconnect it was though).
The frequency aspect of a black start is presumably a bit easier in Europe because there's an interconnected synchronous grid so they can bootstrap it from France essentially.
It's far more problematic for the UK because all the interconnects are DC.
I was recently told by an electrical engineering lecturer that the black start plan here in Ireland is to use the DC interconnectors with the UK to provide startup power to a synchronous generator.
With the new wexford-wales interconnect that went live last month, and another one planned from Cork (?) to France things might be even easier in the near future I reckon.
To me it sounds like an energy company attempting to excuse lack of spending on infrastructure whilst paying out millions to C-suite in bonuses and millions more to shareholders whilst arguing prices have to rise because they can't afford to spend on infrastructure...
Electricity markets and electricity networks are designed by the regulator.
Incentives are planned by the regulator so that individual stations or companies have the correct incentives to have capabilities that the network grid needs.
One example is financial incentives to provide black start capabilities. Another example is incentives to provide power during peak loading (peaker plants). There are many more examples of incentives designed so that the needs of the whole network are met.
If an operator is incentivised to act selfishly in such a way that the grid will fail, then that is a failure of the regulator (not the individual operator).
Blaming individual people or companies for systemic faults is generally a bad thinking habit to form. There are too many examples where I see individuals get blamed. Fixing our systems is hard but casting blame in the wrong places is not helpful. It's difficult to find a good balance between an individual's responsibilities and society's responsibilities.
> Electricity markets and electricity networks are designed by the regulator.
Not quite. They are _influenced_ by the regulators.
And Europe has been incentivizing trash-tier low-quality solar and wind power, by making it easy to sell energy (purely on a per-Joule basis) on the pan-European market.
Meanwhile, there is no centralized capacity market or centralized incentives for black start and grid forming functionality.
> Meanwhile, there is no centralized capacity market or centralized incentives for black start and grid forming functionality.
There absolutely is. Look up terms like "Frequency Containment Reserve" and "automatic Frequency Restoration Reserve". The European energy market takes transport capacity in account, and there is separate day-ahead trading to supply inertia and spare generating capacity. Basically, power plants are being paid to standby, just in case another plant or a transmission line unexpectedly goes offline.
Similarly, grid operators offer contracts for local black start capacity. The technical requirements are fixed, and any party capable of meeting those can bid on it.
It's quite a lucrative market, actually. If during the summer a gas plant is priced out of the market by cheap solar, it can still make quite a bit of money simply by being ready to go at a moment's notice - and they'll make a huge profit if that capacity is actually needed.
No, there isn't. FCR market is not pan-European, and even where it's in place, it's basically in the name only. It's basically only countries that already use rotational generation, so it's not really a stretch for them to participate.
Spain and Portugal are not members, btw.
And the same applies to capacity markets. I believe, there is a plan to come up with a plan for it by 2027.
> Similarly, grid operators offer contracts for local black start capacity. The technical requirements are fixed, and any party capable of meeting those can bid on it.
And I don't believe there are ANY solar/wind plants that have black start capacity in Europe. The current incentives structure makes that a near certainty.
> there is no centralized capacity market or centralized incentives for black start
There certainly is in New Zealand, although the dollar amounts are quite small. If your countries regulator doesn't incentivise the capability, I believe that is a fault of your regulator.
Transpower (NZ) says:
We may enter into black start contracts with parties who can offer the black start service compliant with our technical requirements and the Code. Black start is procured on a firm quantity procurement basis (via a monthly availability fee and/or a single event fee for specified stations). Black start costs are allocated to Transpower as the Grid Owner (see clause 8.56 in the Code for details)
With the DC interconnect, your DC to AC conversion equipment would need the capability to provide synchronized power to the generator you are trying to start. With the synchronized grid tie, your are pulling the generator into the running grid.
A synchronous interconnect provides not just a source of truth but also stabilizes your grid frequency. If you have an isolated grid you have to match generation to demand to keep the grid frequency stable. If you have a 1GW interconnect that means you can mismatch generation and demand by up to a gigawatt and still be fine. I imagine that makes for a much faster startup procedure
You can connect two running grids. Earlier this year the Baltic countries disconnected from the Russian grid, and synchronized and then connected to the European grid.
I imagine you can get close enough by syncing to a shared time source like GPS or the DCF77 signal, as long as you communicate how the phase is supposed to match up to the time source. Or at least you could get close enough that you can then quickly sync the islands the traditional way.
The question is if it's worth the effort and risk. Cold starting a power grid is a once in a lifetime event (at least in Europe, I imagine some grids are less stable) and Spain seems to plan to have everything back up again in 10 hours. Maybe if the entire European grid went down we would attempt something like that by having each country start up on their own, then synchronize and reconnect the European grid over the following week.
Nothing that complicated. You just carefully synchronize the state of the grid on both sides of the interconnect, and when they are perfectly matched, you throw the big switch.
It's a bit difficult in another way: Obviously 50/60hz is not such a high speed that it's difficult to synchronize.
The harder part is this: To pump power into the grid you lead the cycle ever so slightly, as if you were trying to push the cycle to go faster. If instead you lag the phase the grid would be pumping power into you.
That lead is very very small, and probably difficult to measure and synchronize on. I would imagine that when the two grids connect everything jumps just a little as power level equalize, it probably generates a lot of torque and some heat, I would assume it's hard on the generator.
From a physics point of view, by leading the cycle you introduce a tiny voltage difference (squared), divided by the tiny resistance of the entire grid. And that's how many watts (power) you are putting into the grid.
Yes, but that actually makes it harder to start up.
To synchronize the isolated grids, they all need to operate with an exact match of supply and demand. Any grid with an oversupply will run fast, any grid with an undersupply will run slow. When it comes to connecting, the technical source-of-truth doesn't matter: you just need to ensure that there will be a near-zero flow the moment the two are connected - which means both sides must individually be balanced.
And remember: if you are operating a tiny subgrid you have very little control over the load (even a single factory starting up can have a significant impact), and your control over the supply is extremely sluggish. Matching them up can take days, during which each individual subgrid has very little redundancy.
On the other hand, the interconnect essentially acts as a huge buffer. Compared to the small grid being connected, it essentially has infinite source and sink capacity. For practical purposes, it is operating at a fixed speed - any change is averaged out over the entire grid. This makes it way easier to connect an individual power plant (it just has to operate at near-zero load itself, move to meet a fixed frequency target [which is easy because there is no load to resist this change], and after connection take on load as desired) and to reconnect additional load (compared to the whole grid, a city being connected is a rounding error).
If your Factory uses too much power, theres not enough energy to run the power plants generation, which decreases your power production. Death spiraling until theres no power.
You have to disconnect the factory, and independently power your power plants back up until you have enough energy production to connect your factory up again.
Capacitor circuit network warning for alarm for "main grid is drawing on this bank - bad things may happen if capacity is not increased."
Another "trick" is those burner inserters are black start capable. They can pick up fuel and feed themselves to keep running without an electrical network.
I also tend to put Schmitt triggers in low priority areas. They've got a battery on the main grid next to them and if the battery drops below 50% power they remain off until it goes back above 75% power.
That's a mod, not stock. It does two things: feeds the inserter from either end (something I think should be stock--it's a machine that grabs things, why can't it grab fuel it can reach??) and provides a trickle of power to unpowered ones (nice from a standpoint of not having to fuel them when you put them down, but it also adds blackstart capability.
As for 50%/75% triggers--the game doesn't model start/stop problems, only fancy circuit setups would give a hoot about being fed by flickering power. (But as a human....I was out adding to a big accumulator bank at twilight. Far away the bugs had a base close enough to my laser turrets that they kept attacking. The sun was powering my base but didn't have enough for the turrets. The whole bank would flicker for every bug. Usually the electrical indicator on the accumulators is a good thing.)
> The burner inserter is the most basic and slowest type of inserters. It is powered by burning fuel, compared to the more advanced inserters which are powered by electricity. It will add fuel to its own supply if it picks any up, which makes it useful for filling boilers with coal. This has the advantage that it will continue working even if the power fails, as opposed to electrically-powered inserters which will be unable to function.
In particular, when power demand drops below the supply everything starts running slower, which in turn means that the electrical inserters used to feed coal into the boilers run slower which drops the rate of electrical production. Burner inserters don't have that problem.
---
Schmitt triggers are really easy with stock multi combinators now. It isn't so much a "things have issue with flickering power" but rather "turn off the coal feed to the steel furnaces that would go to the electrical station instead" and "make sure that the coal unloading station doesn't brown out" and "turn off power to the electrical furnaces and labs to make sure that the coal mines don't dip in production rate when power dims".
The Schmitt trigger also makes for more reasonable "where is there excess oil that I should produce solid fuel from?" There's the optimal, but sometimes in the processing you can't crack any more crude to gas because the heavy oil is backed up, so turn on the appropriate cracking station when {conditions} and turn it off when {conditions}.
Burner inserters are stock, I didn't realize the fuel leech was now stock. It didn't used to be. And the ability to slowly grab fuel without any source of power is definitely mod. And the mod can leech fuel it's not actually handling--a burner inserter pulling from a furnace will take fuel from the furnace for it's own operation.
I'm just saying what's the need for a gap between on and off? How does the bounce harm anything in the game? Simply put the non-critical stuff behind combinators that will switch off at x% of power. Real world machinery wouldn't like that but the game factories don't care. Nor do the refineries--in your oil case, plonk down a tank of heavy oil, turn on the crackers when it's above a certain level.
> Since v0.10.0, any Fuel items picked up by a burner inserter will also be used to power the inserter. This makes it useful for:
> Automatically loading Gun turrets from a Transport belt, where one side of the belt is filled with magazines and the other with Coal.
> Filling Boilers with Coal. This has the advantage that they will continue working even if the power fails. This is not the case for electrically powered inserters.
> Burner inserter will use item with fuel value for itself when it has empty inventory.
0.10.0 was released 2014.
---
The flickering isn't a problem (though a high frequency flicker can caused the power chart to be difficult to read).
A consistent drain on power that is causing capacitor levels to drop below some threshold suggests that certain things should be turned off before the problem becomes cascading when capacitor supply drops to 0.
Except that in many cases the "consistent" drain is simply night. (Or, day, on Fulgora.) Other than making the chart easier to read what's the case where a 50%/75% trigger is superior to a straight up enable if > 50%?
And apparently I misunderstood the mod, it leeches from things it could pick up but doesn't. It was probably written before it became stock and thus contained some out of date text.
In my server I hooked up a sound alarm to a set of capacitors. Too low of a charge indicates higher power consumption than production, allowing you to unplug certain low priority loads. I also have some emergency coal generators ready to go at the flick of a switch if needed.
Same with Satifactory, The larger powerplants need a lot of energy for their infrastructure to run and an overload will trip breakers and shut the whole grid down, a naively designed grid death spirals very easy. My factory was needing increasingly complicated black start systems so I started putting the powerplant infrastructure on a self contained islands. something a factory overload would not trip, it was something like one coal power plant can run the machinery needed for itself and 8 grid power plants.
Thinking about this makes me wonder how all real world power plants aren't designed to survive a grid outage? Why would you ever need to black start a real power plant? Like, can't you design it so that it disengages from the grid rather than shutting down in a catastrophic outage like this? Or make it only take on a fraction of the load or something?
The problem is that it's still steam pushing spinning rust. You can't instantly scale up or down, let alone do so in an organized fashion. If you were happily generating 1000MW out of 1500MW max, there's very little you can do if a power line goes offline and you're suddenly connected to 2000MW of load - or only 250MW. At best it's going to take tens of seconds to adjust, which in practice means you're forced to dump the entire load because in the meantime your output has deviated so much that you're causing serious damage to downstream equipment - or your own. And load shedding isn't really an option because you're now operating as an island, which means you have to instantly figure out how the grid is now connected, what the current supply/demand is, which neighborhoods should be turned off to best match the pre-incident demand, and what the impact on the local grid is - there are way too many variables there to be able to respond in a fraction of a second.
Starting back up from zero is significantly easier, as you are completely isolated and have zero load. You turn the power plant on, and start slowly adding local load to ramp up. Synchronize with neighboring plants where possible to build the grid back up. The only issue is that a power plant needs a significant amount of power to operate, so you need something to provide power before you can generate power. In most cases you can just piggyback off the grid, but in an isolated black start it means you need a beefy local independent generator setup. That costs money and it's rarely needed, so only a few designated black start plants have them, paid for by the grid as a whole.
Disengage from the grid and do what with the vast amount of power that suddenly has no place to go? That kind of power tends to break things. Powerplants have big cooling towers because disposing of heat is problematic.
And you're assuming that there is a throttle setting that lets the plant produce so little power that it only runs itself. Think of the Falcon 9--the landing is hairy because it's impossible for it to produce less thrust than it weighs. The engine will go out if you try to throttle it too much.
One of the Administrators of the REN, the Portuguese electric supplier is currently giving a press conference. Confirmed they are in scenario of restart from black start.
- Cause of event not known yet.
- They noticed power oscillations from the Spanish grid that tripped safety mechanisms in the Portuguese grid. At the time, due to the cheaper prices, the Portuguese grid was in a state of importing electricity from Spain.
- They are bringing up multiple power systems and the Portuguese grid is able to supply 100% of needs if required. It was not configured in such a state at the moment of event.
- They had to restart the black start more than once, since while starting, noticed instabilities in some sectors that forced them to restart the process.
- Time for full recovery unknown at this time, but it will take at least 24 hours.
We are beginning to recover power in the north and south of the peninsula, which is key to gradually addressing the electricity supply. This process involves the gradual energization of the transmission grid as the generating units are connected.
I see load dropping to zero on that graph, or rather, load data disappears an hour ago.
If the grid frequency goes too far out of range then power stations trip automatically, it's not an explicit decision anyone takes and it doesn't balance load, quite the opposite. A station tripping makes the problem worse as the frequency drops even further as the load gets shared between the remaining stations, which is why grids experience cascading failure. The disconnection into islands is a defense mechanism designed to stop equipment being too badly damaged and to isolate the outage.
Interesting, but in terms of load I think think the data may just be delayed by ~1 hour. Switching to UTC, to avoid timezone confusion, it's currently 13:10:
Everything dropped to zero except wind and solar, which took huge hits but not to zero. I expect those have been disconnected too, as they cannot transmit to the grid without enough thermal plant capacity being online, but if the measurement at some plants of how much they're generating doesn't take into account whether or not they were disconnected upstream they may still be reporting themselves as generating. You can't easily turn off a solar plant after all, just unplug it.
Either that, or they're measuring generation and load that's not on the grid at all.
Probably they are estimates of not grid metered generation assets based on wind speed and solar production, at least in the UK nearly all solar is 'estimated' because it is not measured directly (apart from larger sites), at least in real time.
Rooftop solar for example just shows as a reduction in demand, not 'generation' per se.
This also true for private wind power. Britain has a measurable amount of hill top farms where it just makes good economic sense to install a wind turbine and get free electricity. But we don't meter it, it shows up in charts as an absence - on a windy afternoon maybe Britain is seemingly consuming 4GW less electricity than it "should be". If the wind drops that load reappears on the grid and must be handled by existing infrastructure.
None of this gear is suited to a black start. If you had total grid loss for a month you could doubtless rewire it to power the farm when it's windy despite no grid, maybe even run some battery storage for must-have services like a few lights so they keep working on still days but you could not start the grid from here.
It's not just about the power. System components cannot be brought to operating temperatures, speeds and pressures faster than mechanical tolerances allow. If a thermal plant is cold & dark, it can take days to ramp it to full production.
That's true of some kinds of thermal generators, but not all. Simple cycle gas turbines can come up very quickly (think jet engines). Or your car's engine.
Combined cycle turbines do have a steam component (that's where the word "combined" comes from). The waste heat from the combustion turbine front end is used to make steam in the back end.
Combined cycle turbines are not primarily powered by steam. It is a secondary consequence of their operation. This contrasts with nuclear and coal plants where steam is the prime mover.
The steam part of CC systems shows hours and hours aren't inherently needed to get a steam power plant in operation. For that matter, warships with steam propulsion show the same thing, I believe.
A true black start has several factors (which make it difficult and notable):
1. The grid has to fully collapse with no possibility of being rescued by interconnection
2. As a result, a generation asset has to be started without external power or a grid frequency to synch to
3. An asset capable of this is usually a small one connected to a lower voltage network that has to then backfeed the higher voltage one
4. Due to the difficulty of balancing supply/demand during the process, the frequency can fluctuate violently with a high risk of tripping the system offline again
None of this applies in yesterday's case:
The rest of the European synchronous grid is working just fine.
News reports stated Spain restored power by reconnecting to France and Morocco.
By reestablishing the HV network first, they can directly restart the largest generation asset with normal procedures.
As they bring more and more load or generation online, there's little risk of big frequency fluctuations because the wider grid can absorb that.
Just to add, I was at a University campus when the entire building's electrics went out and there was a significant pull due to relatively powerful computers in every room. They initially tried to bring the building back online altogether and failed. Then they tried to bring it back in sections and failed too. In the end they ended up going into each lab, turning every computer off at the wall to bring each lab's power back, and then turn each computer on one by one.
I can only imagine the difficulty of bringing large parts of the grid back online, that rush current must be immense.
Yup. I used to work with a factory that had a bunch of really big machines. Turn everything on at once and the transformer out on the pole self-destructed. Note that the breakers didn't pop--the startup transient was short enough. The power company wasn't happy.
Or look at Apollo 13. The astronauts had turned off everything possible because they had lost their generator and only had their batteries. And it took a lot of furious planning by the guys on the ground to come up with a sequence of turning things back on that didn't cause the peak draw to go too high. Can't go too fast or it trips. Can't start too early because the power is limited, but can't start too late because the systems have to be up when they hit the atmosphere.
I did a project where we predicted transformer failure (they blow up!) from changes in the oil that they have in them (its insulative properties suddenly change). This was 30 years ago so it's all a bit fuzzy, but the one thing that really stuck with me was the story that the SME that we were working with had about the UK grid teetering on the edge of failure during the "great storm" of 1987. His telling was that they really were unsure if they would get it back at all!
> It's one reason it's a good idea to always have some cash at home.
More than cash it was important yesterday to have the following in case it would have lasted longer:
- a battery powered am/fm radio with spare new batteries
- some candles and matches
- food reserves for a few days that don't need refrigeration: bread, anything in can, pasta, rice...
- some kind of gaz or alcohol stove, dry wood or bbq charcoal: you can always make a fire in the middle of the street where there is no risk of burning things around.
- water reserve (I always have like 24L of drinking water) and since I hate waste I regularly fill jerrycans when waiting for hot water in the shower that I use for manual washes (kitchenware
or gears).
Does solar power make this process easier or harder? I know that with thermal plants you have a spinning mass that you have to synchronize, and phase shift is used to assess how hard the plant is working (and whether to trip a disconnect as we see here)
But with solar, how is the synchronization provided? In like a giant buck? Or in software somehow? Does the phase shift matter as much as in the electromechanical systems?
My intuition is that solar would make the grid harder to keep stable (smaller mass spinning in sync) but also may offer more knobs to control things (big DC source that you can toggle on/off instantly.. as long as sun is out). But I don’t actually know.
Mike_hearn's comment was grey but was correct: phase following is indeed done through software in the inverter. Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat.
Currently the main driver of battery deployments is not so much energy price time arbitrage as "fast frequency fresponse": you can get paid for providing battery stabilization to the grid.
So if you have a smarter solar panel, or a smart battery, you can stabilize the grid. I’m assuming that all of the traditional software complexity things in distributed systems apply here: you want something a little bit smart, to gain efficiency benefits, but not too smart, to gain robustness benefits.
My intuition is that bringing the market into it at small timescales probably greatly increases the efficiency significantly but at the cost of robustness (California learned this “the hard way” with Enron)
> Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat
If the electronic controller is “ahead of” (leading) the grid, then that heat would come from the solar plant; if it is “behind” (following) then that heat comes from the grid. Is that right? And likely, solar plants opted for the simplest thing, which is to always follow, that way they never need to worry about managing the heat or stability or any of it.
I wonder if the simplest thing would be for large solar plants to just have a gigantic flywheel on site that could be brought up via diesel generators at night…
Most solar and wind plants follow the inertial lead of the thermal plants. They can't synchronize without enough thermal generation being online. Supposedly there are efforts to change that, I don't know enough about grid engineering to say how far along that might be in Spain.
> But with solar, how is the synchronization provided? In like a giant buck? Or in software somehow? Does the phase shift matter as much as in the electromechanical systems?
If you mean how does solar act to reinforce the grid: search for terms like "grid forming inverter vs. grid following inverter" though not all generators are the same in terms of how much resilience they add to the grid, esp. w.r.t. the inertia they do or do not add. See e.g. https://www.greentechmedia.com/squared/dispatches-from-the-g...
Harder mostly, See the frequency is set by huge rotating masses in the form of generators, and when the supply and demand is matched the frequency and voltage are stable, when demand dramatically increases it pulls the frequency and voltage down, which is effectively slowing the generators down as load / magnetic drag increases with current drawn. Having large inertial masses spinning actually helps smooth out frequency changes. whilst large solar farms can and do syncronise with the grid, they are reactive and do not add the same smoothing effect as humungous spinning masses.
Low Grid frequency & voltage can cause an increase in current & heating of transmission lines and conductors and can damage the expensive things, this is why these systems trip out automatically at low frequency or low voltage, and why load shedding is necessary
> Harder mostly, See the frequency is set by huge rotating masses in the form of generators
I'm not saying you're wrong, but this isn't obviously correct to me.
Since solar going to a grid is completely dependent upon electronic DC->AC conversion, I would expect that it could follow a lot greater frequency deviation for a lot longer than a mechanical system that will literally rip itself apart on desync.
The real reason that small scale solar PV is grid following (i.e. it depends on an external voltage and frequency reference) is that this ensures power line safety during a power outage. That's it.
An inverter can be programmed to start in the absence of an external reference and it can operate at a wide range of frequencies.
About 10 years ago I had a chance to work in the Utilities vertical for a power producer. I asked the same question: how is the frequency set and the answer was that the biggest power plant sets the frequency because they can produce a lot power and any other smaller generator would not be powerful enough to change the frequency.
Sure, the smaller plant gets pulled into the larger plant's inertia.
However, DC-AC converters don't have an inherent inertia. They can follow almost any frequency and phase within reason. Certainly a DC-AC converter should be able to respond way faster than any frequency/phase changes that a mechanical system can generate.
In theory, they should be able to set themselves to be ever so slightly closer to ideal so that the amount of power they have to sink is limited but are still exerting a very slight force to bring the grid back into compliance rather than continuing to add load which propagates the collapse.
I'd say a little harder to negligible now, but potentially way easier in the future.
The main difficulty is that the software of grid-following inverters tend to make them trip out very suddenly if the grid parameters get too far out of spec (they will only follow the grid so far), but once the grid is good they basically instantly synchronise.
But all large solar farms are likely to be mandated to switch from grid following to grid forming inverters eventually which will make them beneficial for grid security because they will help provide 'virtual intertia' that looks exactly the same to the rest of the grid as spinning mass does.
As the press release you linked points out, the black start plan Spain trains on uses nuclear energy supplied from France to re-energize the power plants.
"Luckily", France is at an historically high level of production capacity at the moment and the connection between the two countries was reestablished fast.
According to RTE (French network manager), the interconnection was maxed yesterday at around 3GW of power.
Sadly, while Spain is part of CESA, it's not very well connected. I wouldn't be surprised if one the takeaway from the whole incident is that more interconnections are needed.
Ukraine is interesting in this context because there are so many generators. In the richer parts of Odesa I've even found it hard sometimes to tell whether or not the grid power is on as literally every single building had sufficient backup generators to keep the lights on (also, many big businesses seemed to run their generators even when the power was on, I presume as a civic-minded way to add generation capacity overall and avoid interruptions when the power flips on and off).
Generators are not in sync with the grid though. And that's the hardest part.
Here's why generators were running here despite the grid being available. A generator has a very short lifetime and in order to prolong it, some owners learned to run those in the very optimal schedule. Which sometimes requires a minimum amount of time to run in a single cycle. Thus if you started it you are committed to run for X hours.
> Generators are not in sync with the grid though. And that's the hardest part.
I know. What's interesting re: Ukraine is because there are so many generators there are more options to getting power sufficiently restored for normal life than just rushing to restore the entire grid.
> Thus if you started it you are committed to run for X hours.
I'm skeptical this is the main reason. While fast starting a diesel generator is hard on it, there are other, slower, ways to start big diesel generators with minimal impact on lifespan. The blackouts in Ukraine are almost all on a schedule, so big buildings with dedicated staff and expensive generators can and do startup their generators in advance (I've personally heard this happen on occasion - big generators spooling up multiple minutes in advance of the scheduled blackout).
Also, it makes sense to turn on generators in advance anyway: gives you a chance to diagnose any issues.
Perhaps not the main reason, but heavy devices do indeed require minimum amount of time running. I think small devices as well but not as long. Honestly I don't remember what we did back then: so many things were going at the same time, longevity of generators was not very well know at the time )
And if you are a mere mortal in this world, play Factorio : Space age expansion on the planet Aquilo. To learn the precious importance of reliable multi stage power bootstrapping
Power never went out in a country completely. At the lowest point consumption was ~40% of normal for that time of day.
Ukraine went through many black starts in the first winter of Russian strikes against energy. I guess they built a skill of recovering it quickly enough that it started happening faster and easier every next time.
I would think that renewable infrastructure could be the fix, at least if you start installing larger battery capacity to meet renewable store and usage shifts, the grid essentially is installing the resources that can also be used to respond to & contain sudden source losses and prevent cascades.
I wonder if someone could build a realistic scenario into a game -- let's say some sort of smaller scale black start for, say, a space station. And throw in a unknown computer architecture for the in-game computers so that players need to RTFM to figure out how it works.
I took down the servers though, so you probably can't easily try it. I don't know if I added a way to configure the lobby server. I should have! It's open source though. And there is a video about that thing on my YouTube: https://www.youtube.com/watch?v=6TPgfa7LbiI
The game is bad and nothing of what we planned on doing actually made it into the game. The video is long and boring too. But maybe someone finds this cool and is inspired by this and makes a game like this.
The first 15 minutes of the game were actually about getting the ship moving, first by reading the manuals of half a dozen different ship systems and then following some procedure outlined in those manuals (parts of which were simply incorrect), maybe having to do some things in sync with your other players and stuff like that. I think it would have been cool to add multiple reactors and start them up in sync and stuff. The different ship systems were actually Lua programs that interacted via a message bus. So kind of a unknown computer architecture?
For maybe the first 24 hours at a grocery store, and then not so sure. Would your neighbors sell you supplies and food? Maybe not? And so many places now depend on cashless transactions and doubtful they have pen, paper, lockbox, and safe as a contingency plan.
It would be essentially the same thing as a grid black start, except that the first breaker to close has the European grid on its primary side, instead of a freshly started generator under your control.
The complex process of configuring the transmission network to bring grid power to each power plant in succession is the same.
The continental part of the EU runs on one synchronized grid. The Nordics (except half of Denmark and, uh, Iceland, Greenland etc) runs on a separate synchronized grid.
I'm confused. Would the start ever have to be truly black? Wouldn't water always be driving hydroelectric turbines, generating some electricity? Solar panels generate electricity without requiring input. I understand that synchronizing AC is not trivial, I'm only questioning the part about whether the start is truly black.
The tills have keys to manually open them, and you can just record transactions on pen-and-paper if needed and enter them later. I've seen plenty of businesses do exactly that during power outages. Not as fast. But totally doable.
Stores with tills and freezers etc will have power for the tills but the backing network for payments probably won't be up. That's the concern with being cashless. They can accept cashxl, but no one has any.
I was able to buy some groceries and pay with card. The tills had a battery backup and the network infrastructure that supports card payments was apparently working.
That said, lots of people hit the cafés and had to resource to cash payments. There was also lots of people buying bottled water at the shops.
So basically, you could divide people in two groups. Those that took it like an extra Sunday, and those that took it like the beginning of a war or something :-D
Most people around here seemed to be balanced. Yes, a lot of people outside at bars/cafes enjoying a spare Sunday, but also preparing quietly and slowly in case it takes longer time. All the stores ran out of batteries, radios and such, at least the ones we visited, and I only managed to make one card payment around 13:30 sometime, after that internet stopped working 100% until the night. Almost no super markets were open around here, maybe one or two who had generators it seems.
>It's one reason it's a good idea to always have some cash at home.
Most places are so dependent upon electricity that they can't even take cash during a blackout. And they don't even have the mechanical machines to take a credit card imprint anymore.
The last of my raised number credit cards went away last month. Those old machines will only get a blank rectangle from me. Sad, because I did actually use one of those about 3 years ago when a rural gas station had power but no network.
Network is not required for mag-strip or emv operations, but the terminal has to be configured accordingly (and has to have some way to send batch of transactions for settling). It's less common now, but fully supported.
I actually have no real clue how and where Spain/Portugal is connected to rest of Europe but could they also restart North to south with help from the grid from France?
The grid uses AC (not DC), running at 50 Hz (cycles per second). So the voltage is going up and down at that frequency, in a sine-wave pattern.
If you try to connect another generator to the grid, it needs to be at the same point (phase) in the sine-wave cycle, so that its power contribution is added, not subtracted.
If it's not in sync, huge currents can flow, causing damage. Sort of like connecting jumper cables backwards.
The bulk of the power grid is alternating current (AC), and the frequency of the resulting sine wave needs to be synchronized with the other parts of the grid it is connected to.
I'm going with: never attribute to malice what can be explained by ... an incredibly complex system that can fall over even if no-one's being stupid. I would want very strong evidence before I believe this is an attack.
I remember the day when the Swiss railway power network went down for a day (in 2005) because one power line was down for maintenance and someone pressed the wrong button and produced a short circuit somewhere else. It's a bit like the incidents in planes were one engine has a problem and the crew shut down the other one by mistake.
> The animal had come into contact with the transformer at the station, disrupting supply to the entire country. There were no immediate details on whether the monkey survived the incident.
I'm sure the constituent atoms of the monkey survived.
I saw a video the other day of some human running and jumping on a transformer after hopping a fence, dancing on the transformer in a distribution site.
It ended as you'd expect, a bright light, a lot of curse words from the camera operator who was probably blinded temporarily.
If that's all it took to turn Srilanka off.Even if that country lags behind western Europe, what would it take for a few humans who intentionally cut off power.
Trying to stay on the facts, this incident is likely accidental but some people even the very workers at energy companies could send a message for, I don't know. A pay raise?
Isn't it amazing? In movies, evil terrorists take out the grid all the time (or threaten to), to terrible consequences, and they need genius hackers to do so. In reality, it's easy peasy and still, people hardly ever do it (and nobody (ish) dies if they do). Humans are good!
Yes, you can try to hold the country hostage for your salary by going on strike, but that's the sort of thing that results in very energetic union-busting.
Actually sabotaging the infrastructure would result in terrorism charges, or at the very least the JSO treatment.
I mean i'm pretty sure there are laws in place everywhere to avoid a single pay raise strike doing nation wide catastrophic events. Hospitals for example are usually limited in how they can strike.
That's assuming the people who want to strike care about having a job afterward. I know the state hospitals stop loss people when there's a disaster, so I keep abreast of predictable disasters and tell my wife and friends "hey you're sick, go get a doctor's note for the rest of the week" and then they're not stuck at the hospitals for days during a disaster.
What kind of person fakes illness expressly so they don't have to help out during an actual disaster when their skills are needed? "Whenever I hear there's a fire, I tell my firefighter friends to take the battery out of their pager so they won't have to go fight the fire." I agree we owe these people better treatment than just mandating they work heroic hours, but this only makes things worse on those who comply with the mandate.
My neighborhood and community is more important than the hospital; the friends i mentioned specifically are also radio operators, so they have more important things to do during a disaster than be a warm body at not that type of hospital. A sanitarium.
The low-wage workers can get triple-time and take weeks off "later in the year" for covering the stop-loss, so they also benefit.
I don't call fire fighters, because they are emergency responders. Please note i said what i do, not what i think other people should do. The people i tell this to are free to do with that information what they want, i don't demand it or anything.
Consider this a contextual error on my part. It's more of an "inside baseball" snippet of conversation than anything that requires judgement.
That wasn't tree falling in Ohio, that was overloaded line sagged and shorted into a tree, compounded with several other factors that contributed to the grid instability and the inability of the grid operator to realize how unstable the grid was.
I think the point is that neither of those events should take out a whole city; the design is such that there is considerable redundancy in the system.
Indeed. For instance, power grids ideally operate with N-1/N+1 redundancy, i.e., the disablement of any single power line should not cause a cascading failure.
Relying on vegetation management along millions(?) of miles of lines is the definition of thinking you're in control without even knowing what risks exist. I see this every year in Texas, they trim back trees thinking they have solved the problem - except - they are allowed to trim them back I think 6-8 feet from the lines, ignoring the fact the trees are 20-40 feet taller than the lines. Nobody seems to see the risk that they left there but it's quite obvious to me, we're not eliminating risks at all. We're minimizing them at best, and even then the labor required to trim all the branches around lines every year is impossible to keep up with from an economic feasibility standpoint.
They trim back what poses the highest risk with the budget they have.
As you have identified A wider right of way costs more.
Usually for lines above some voltage, perhaps 200kV, the cost of an outage due to a tree strike outweighs the cost of additional vegetation management so they will clear the right of way wide enough that no tree can fall and hit the power line.
Around here for 130kV the right of way is still as narrow as it can be and we annually take down the riskiest trees as this is the best for our budget, which is not unlimited.
Concur, they have huge fire breaks they put high tension lines in, where the easement is easily 100' to either side of the lines. This is in Louisiana. When there's a hurricane these lines still break, but not because of trees. During heavy storms, smaller trees hit smaller (480? Some KV?) Lines that go along highways to residences. A high tension line down means a few days without power (about 3-4), a tree on a lower voltage line is usually fixed within hours.
They go through and remove damaged trees near the easements of the highway lines, as well as branches that could break into lines.
As an aside we lived on the same section of grid as the sheriff, and our power was rock solid for a few years, then he left office and now our power is better than average (at least better than our neighbors who's power line cones from the other direction).
Where I live, there are no trees in the path of high-voltage (400KV) transmission lines. Everything is cleared below them and the voltage lines are about 90-120ft high so even if some trees grow, they will not touch the lines.
I vaguely remember an episode of some thriller series when I was too young to be watching it, perhaps Twilight Zone (or similar) where someone was hearing screaming, they went out a d saw trees being cut.
Having lived through the Texas electricity fiasco in 2021, I would blame cost cutting, the reckless drive for “efficiency”, and maximizing shareholder value.
In Texas, the electric providers cut staff and maintenance to maximize shareholder value. They will not have redundant systems and redundant plants out of the goodness of their hearts. The Texas marketplace actually allowed them in the odd event of an outage to charge astronomical spot prices thinking this will incentivize them to have redundant systems. This was a foolish fantasy.
Probably didn't help that before the outage hit, Spain was running its grid with very little dispatchable spinning generation, and therefore not much inertia.
This is (a) incredibly impressive to achieve and (b) definitely the point at which the battery infrastructure needs to catch up in order to reduce the risk of such incidents.
Just to clarify this a little bit… flywheels are cool for absorbing transients and providing short-term hold-over during a blip but don’t have great long-term capacity.
Spinning reserve in the grid is equipment that capable of long-term generation very quickly. In the case of hydroelectric dams, they will often cut off the water supply to some of the turbines and use air pressure to push the water out of the way; the generator attached to the turning essentially turns into a motor and keeps the turbine spinning. If you need to bring it online, you open the water valve and let the air out.
Similar situation with natural gas-fired simple cycle turbines. They’re sitting there running at low output. Need more? Just add fuel. For combined cycle it might take a bit for the boiler to warm up for full output but having the first stage running full tilt will get it warmed up fast.
In electrical terms 4 GJ is 1,111 kWh. That's about 15 EV batteries' worth, or about £220 worth of retail electricity (at 20p/kWh). So it's a lot compared to the usual domestic things people are used to, but not much in grid terms.
Or if you consider the Irish grid (average consumption around 5 GW) that's enough energy to power the grid for about 0.8 seconds (obviously it's not going to have enough instantaneous power output to do that, but again for a sense of scale).
If Ireland had 10 of them, that'd be 8 grid-seconds worth of energy. Although, of course, actual disturbances aren't going to be that large. A few percent imbalance perhaps?
So if the whole grid had an instantaneous 10% imbalance, one of those units could carry it for 8 seconds.
(EDIT: changed energy numbers to fit the appropriate power grid)
flywheels don't really work synchronously, though. Or at least, if so, they're not very useful as storage. Inverters can simulate inertia just fine, in fact they can simulate a much larger inertia than the corresponding power of spinning generation, leading to greater grid stability (whether connected to batteries or solar/wind, though without batteries its a little 'brittle' if you run near 100% output, as the power needed may suddenly disappear, so this is probably not good practice for the system as a whole. And you still need some give in your inertia so the grid can actually communicate about supply and demand)
They're usually almost more capacitators than they are batteries from an electrical standpoint. They're there to rapidly smooth out the smaller jitters.
Spring is nowadays the best time for the grid and renewables. Low consumption as heating period is coming to and end, air conditioning is not required yet and the sun can be strong already providing lost if solar power output. I can imagine with a grid and a huge growth of solar generation that there are a myriad tripping stones that can upset the balance.
Might as well mean that they suspect the culprit, but they're not going to do anything anyway for this specific culprit, so let's all pretend it was an accident. Why make yourself look like a fool?
They already became a laughing stock once for promising the "strongest possible response" for the Nord Stream 2 sabotage [1].
> but I thought they were incredibly hardened, with backups and contingencies in depth
Some are harder than others, and some have random flaws which nobody can really predict.
Spain seems in the transition to renewables, so it's possible that they have some flaws because they are still in the process, or because it's something which never happened before and is unknown territory. Also, Spain had some economic problems in the last decade, maybe someone build to cheap or was even cheating somewhere.
> Are the grids at this scale really this brittle? Would there be a death toll from this?
Hospitals should have backup-systems. Traffic should be able to stop in time. I guess the most problematic parts are people stuck in elevators and other spaces which only open electrical, as also the loss of cellular phone-connections for calling helpers.
> the loss of cellular phone-connections for calling helpers.
All the mobile phone installations that I saw had power for at least 24-72hrs depending on how far from civilization they were. The carriers have backups and everything.
The problem in these kind of situations is the saturation of the mobile network, not its availability.
Hardening focuses first on not damaging equipment and second on providing energy. If things go wrong quickly enough you don't have time to react, because after a power plant disconnects you get sudden bumps in load that can trigger a chain failure near the original point. The last time it happened in Europe was in 2003, which isn't too shoddy.
What Spain's PM is saying (and is being reported by Spanish newspapers), 15 gigawatts of energy production went down all in 5 seconds. Hardening to tolerate that much of a change, that fast is a more extreme event than a grid the size of Spain's preps for.
To be fair, a big deal of those 15GW probably went down as a result of the initial outage: equipment is designed to intentionally disconnect completely when the grid is behaving badly, rather than trying to force it into submission and potentially causing serious equipment damage.
In reality this means you might lose, say, 1GW due to a transmission line failing, have a big frequency dip as a result, and then have 14GW drop offline like dominoes because they sense a grid frequency outside of safe operating parameters; disconnect as they go into safety mode; cause the frequency dip to worsen; and pull even more plants offline with them. If you're not careful, a small outage can quickly cascade into an entire grid going offline.
> I'm going with: never attribute to malice what can be explained by ... an incredibly complex system that can fall over even if no-one's being stupid. I would want very strong evidence before I believe this is an attack.
I think we should prepare for the worst though. It's wrong to assume it's not an attack too, and until we can conclude it's not an attack we should be prepared to deal with the possible consequences and act accordingly.
I think James Burke's classic talking about the fragility of our complex interdependent systems starting the episode from the 1965 Northeast blackout is still relevant and an interesting watch: https://www.youtube.com/watch?v=XetplHcM7aQ
Every single retrospective I've seen on major power outages is full of miscommunication and people who would be fired if they worked in fast food. As well as blaming the tech boogeyman ("Our systems were on the fritz this morning")
A few months ago I was hit by a blackout literally the second I was about to start delivering a company-wide talk on AI. Everything went out - Internet, mobile networks, street lighting, the lot.
We're a remote business so it seemed like I'd just rudely dropped off the call, but as everything was down I couldn't let people know what'd happened.
Apparently it was caused by botched maintenance work affecting 30,000 houses, but the timing was so perfect I can't help thinking it was because our AGI overlords really didn't want me to deliver that talk for some reason.
I don't think we're able to tell from the data if one is the cause of the other, are we? Since if production was lost, load would have to be shedded to balance the grid, and if load was lost (e.g. due to a transmission failure), production would have to be disconnected to balance the grid.
That started from a combination of a lightning strike and generator trip, but turned into a local cascade failure as lots of distributed generation noticed that the frequency was under 49Hz and disconnected itself. I suspect the Spanish situation will be similar - inability to properly contain a frequency excursion, resulting in widespread generator trips.
(I suspect this is going to restart a whole bunch of acrimony about existing pain points like grid maintenance, renewables, domestic solar, and so on, probably with the usual suspects popping up to blame renewables)
> probably with the usual suspects popping up to blame renewables
Renewables were a factor in the blackout here in Brazil a couple of years ago: the models used by the system operator did not correspond to reality, many solar and wind power plants disconnected on grid disturbances quicker than specified. That mismatch led the system operator to allow a grid configuration where a single fault could lead to a cascade (more power was allowed through a power line than could be redistributed safely if that power line shut off for any reason), and that single fault happened when a protection mechanism misbehaved and disconnected that power line. The main fix was to model these solar and wind power plants more conservatively (pending a more detailed review of their real-life behavior and the corresponding update of the models), which allowed them to correctly limit the power going through these power lines.
The problem is that some media have a tendency to present "operators did not properly model renewables" as "renewables to blame for blackout". The cause is poor modeling, but renewables get the blame as it reinforces a pre-existing "renewables bad" mentality.
This is why solar requires redundant data links to the operator in CA now. Mine has it. During instability they can command my system to accept larger voltage and frequency excursions than would normally be allowed to prevent a "pile-on" where an excursion causes solar to make the problem worse.
If you have a switching power supply brick for your tiny USB device, not really.
If you have a large spinning inertial mass like a factory motor or a power generation turbine, it's extremely important. Imagine a manual car transmission, but there's no slip-clutch, you need to perfectly align engine with the wheels rotating at 300mph, and the inertial mass you're up against if it's not perfectly synchronized is a freight train.
That's why generators trip offline in a blackout cascade if the frequency deviates out of spec. The alternative is your turbine turns into a pile of very expensive shiny scrap metal.
Frequency is effectively more important than voltage. Or rather it collapses first. Frequency reduction for a rotary generator means that more energy is being taken out than the rotary shaft energy is being put in, so it is effectively an early warning that voltage is about to drop.
Frequency coordination is absolutely critical, via phase coordination. A large generator must not get significantly out of phase. So frequency going out of spec triggers the generator to "trip" (disconnect).
As I understand it, it's not possible for a generator to get out of phase with the grid. Sudden changes in the grid phase instantly and catastrophically break generators, power lines, transformers, and so on, but that can only occur when you connect two grids together without synchronizing them. Which implies there are two separate grids to connect.
I don't know what specific threat is addressed by tripping generators offline when the frequency deviates by 1 Hz. Are they so mechanically fragile that is already damaging to them, or is it a precautionary measure because that kind of instability is likely to precede sudden frequency or phase jumps that are damaging?
> Are they so mechanically fragile that is already damaging to them, or is it a precautionary measure because that kind of instability is likely to precede sudden frequency or phase jumps that are damaging?
Both actually. A frequency mismatch between what the grid has and what the turbine is supplying causes significant thermal losses, so you got to trip the generator anyway rather sooner than later, but a significant frequency deviation is always a warning sign that something is Massively Broken and requires immediate attention to find the cause - too low a frequency means you need to shed load immediately, too high means you need to shed generator capacity immediately.
Yeah, I think it really helps to think about the grid as basically one giant spinning machine. Everything is connected with wires instead of driveshafts, obviously, but a low frequency means that all of those machines that are burning fuel to try to spin at the right speed can’t generate enough torque to spin the rest of the machine at the right speed. And on the other side, spinning too fast means that all of the machines are burning too much fuel and need to slow the heck down right now (which can be hard to do with, for example, nuclear).
That's the point though? The generator can't be lower or higher frequency than the grid. It can be applying its maximum power into a grid that is still short of power, which is probably a better situation than if it was not doing that. Tripping a generator that is supplying power into a grid with too much power (frequency too high) at least makes logical sense. Shutting down generators on a grid with too little power (too low frequency) makes no sense, unless the generator itself risks damage. Loads should be tripped offline when there's too little power, not generators.
One of the parent posts correctly identified there are thermal issues with operating at significantly off nominal frequencies for extended time periods. System operators will specify frequency bands and minimum time a generator has to stay online for each band. For example we have to stay online indefinitely for frequencies 58.5 to 61 Hz, and can trip after 60 seconds for 57.5 to 58.5 Hz, and can trip instantly below 57.5 Hz.
Some plants the system operator has the ability to send a signal to instantly open the generator breakers to shed generation if necessary.
>> It can be applying its maximum power into a grid that is still short of power, which is probably a better situation than if it was not doing that.
> One of the parent posts correctly identified there are thermal issues with operating at significantly off nominal frequencies for extended time periods
Yeah, that's very much the problem. Power sources that burn fuel to generate electricity generally don't like being run at 100% throttle for long periods of time. In a low frequency situation you've got potentially multiple countries of generators all running flat out trying to get the frequency back to where it's supposed to be. If they reduce throttle without having something else come online to take the place of the energy they were producing, the line frequency drops further.
Nuclear has its own issue with running full throttle: once a plant has been running at 100%, it takes time for it to throttle back down and might not be possible to immediately throttle back up. There's decay heat from the fission products and there's short-ish lived (up to 40-50 hours though) neutron poisons.
There is no such thing as a frequency mismatch between the grid and the generator. There are frequency deviations from 60 Hz but turbine and grid deviate together
Yes. The speed of AC electrical machinery is determined by the frequency and number of poles. This means generators connected to each other in parallel MUST rotate in synchronism. If the frequency changes the generator is now producing power out of phase which can cause a fault (a short circuit) and possibly damage the machinery.
Its like a three legged race. You and your partner have to run in synchronism. If either of you slows down or speeds up, the other can trip and fall over taking both of you out.
Frequency is health of the grid. If it's an entire hz off of where it should be something is massively wrong. Most generating equipment will trip on a deviation that bad.
What I wanna know is if I could/should have a plug-in box that detects the frequency is drifting and the grid is unstable —-> high risk of imminent failure.
Might even give you clues about something big tripping offline.
Probably lots of false alarms, but if it an outage is particularly bad for you, good to know as soon as the system operators do.
Obviously over the internet could work too, but who wouldn’t love their own box?
> Probably lots of false alarms, but if it an outage is particularly bad for you, good to know as soon as the system operators do.
When the phase gets pulled down hard like what nearly happened in Texas and what probably happened here, it'll go from looking like the background noise of phase changes to catastrophic in just a few seconds. It isn't like you'll get warning an hour ahead of time. You'll probably notice your computer monitor going dark before your grafana graph refreshes.
Old school electric clocks used to (still?) keep time with the grid frequency. Guaranteed to always have N cycles per day. So if things are gradually failing, you might be able to see your clock keeping worse time.
That's not going to work, because the clock will mainly be showing phase drift.
Having the grid operate at 49.99Hz instead of a perfect 50.00Hz for a day will make your clock lose 17 seconds, but it's completely harmless. That's normal regulation, not a gradual failure. The grid chooses to compensate for that by running at 50.01Hz for a day, but that's solely for the benefit of those people with old-school clocks - the grid itself couldn't care less.
A failure means the frequency drops from 50.23Hz to 48.03Hz, probably within a single second. You'd notice as your clock stops ticking due to the resulting power outage.
if a piano is out of tune, you'll hear it, perhaps even flinch.
if a high power transformer goes out of tune... it melts or blows up (or both); it'll try to shut itself down before that. getting it back on becomes a problem if other transformers do the same thing, which is apparently what happened in the whole country.
Wow that's interesting, solar was absolutely peak at the time. Maybe related to poor frequency leading from solar powerplants coupled with some other spike that drew it out of sync?
Interesting - when you realize that nuclear energy is not as reliable as solar (nuclear disappears completely around 1 pm, while solar keeps a stable input share).
Looks like they're using French nuclear to help black start the grid [1]. Solar and wind systems could have caused the instability issues that led to the Spanish Nuclear requiring isolation. It will be interesting to see the final investigation, but my bet is that "induced atmospheric vibration" is a PR deflection from a badly designed and operated system [2].
[1] https://transparency.entsoe.eu/transmission-domain/physicalF...
[2] I'm not necessarily blaming the engineers, but the politicians who force those engineers to put square pegs in round holes. For example, I can imagine politicians making a short term decision to skimp on energy storage while increasing renewable penetration. Surely renewable systems must be less reliable without storage given the lack of rotational inertia?
According to [1], 3/4 came from renewables during the re-startup. Most of today's inverters are black-start ready. That is, they don't require external energy to startup, unlike nuclear, which is heavy, costly, and slow. German readers may find fefe interesting here [2]. He debunks many of these myths.
Apparently a local grid overload near France and a cascading failure down the Spanish network, but radio and newspapers don’t agree on root cause. Of course there is a lot of noise.
For instance, one reporter asked one of the government flunkies whether it could be a cyberattack and they turned his noncommittal “maybe, we don’t know” into “government says cyberattack may be ongoing”.
Be careful of idiot reporters out there.
Edit: I’m listening to another radio interview where they are outlining the plans to bring online Portuguese dams and thermal generators over the next few hours, progressively unplugging from the Spanish supply (fortunately we have enough of those, apparently).
It should take 3-4 hours to get everything balanced with only national supplies, and they will restore power from North to South.
Key points that started it were (you can see the chain of events in the doc):
2.4.1. At 16:52:33 on Friday 9 August 2019, a lightning strike caused a fault on the Eaton Socon – Wymondley 400kV line. This is not unusual and was rectified
within 80 milliseconds (ms)
2.4.2. The fault affected the local distribution networks and approximately 150MW of distributed generation disconnected from the networks or ‘tripped off’ due to a safety mechanism known as vector shift protection
2.4.3. The voltage control system at the Hornsea 1 offshore wind farm did not
respond to the impact of the fault on the transmission system as expected and
became unstable. Hornsea 1 rapidly reduced its power generation or ‘deloaded’
from 799MW to 62MW (a reduction of 737MW).
Curious question for someone familiar with power at grid-scale -- How granular is load shedding? And how is this measured / tracked?
In my head, I'm thinking of generators/plants, connected by some number of lines, to some amount of load, where there are limited disconnection points on the lines.
So how do grid operators know what amount of load will be cut if they disconnect point A123 (and the demand behind it) vs point B456?
Is this done sort-of-blind? Or is there continual measurement? (e.g. there's XYZ MW of load behind A123 as of 2:36pm)
I can speak for the GB case. Low Frequency Demand Disconnection (LFDD) occurs automatically and in stages when the frequency drops until it stabilises. The substations or feeders that are tripped off are not currently determined by real-time metering - instead they are pre-allocated based on their typical demand. This means that the system operator does not really know how much demand will be disconnected at any given time. If it's sunny, you could easily trip off a lot of solar generation connected on the low voltage network, causing the frequency to drop further. It is far from optimal!
> The substations or feeders that are tripped off are not currently determined by real-time metering - instead they are pre-allocated based on their typical demand. This means that the system operator does not really know how much demand will be disconnected at any given time.
This is wild. From a amateur technical perspective, it would only take a cheap hall sensor inside the transformer to have a pretty good guess of how much current has been flowing to the load.
Hell, put the hall sensor onto a board with a micro controller and a LORA transmitter and stick it to the outside of the feed line. Seems like an incredibly cheap upgrade to get real-time load data from every substation.
The nice thing about frequency based regulation is it's an inherent property of the system, so as long as you're connected to the grid you've got the info you need to decide when to turn on or off.
If you're monitoring real time power consumption you then need a whole extra infrastructure to communicate this info back and forth. Of course you then have to consider how you're going to keep that extra infra online in the event of power issues.
Frequency based regulation is only telling you that something is wrong, but not what or how to fix it.
If you find yourself in the middle of a black swan event, and 15 GW have tripped offline, you have milliseconds to dump pretty much exactly 15 GW of load, otherwise more generating capacity is going to trip offline very quickly.
If you only dump 14 GW because you used historical data (which happens to be imprecise, because today's cloud cover reduced rooftop solar output), you're still going to be in trouble. A detector scheme with sensors at every substation would allow you to do just that.
The board is not the expensive part. It's the getting reliability qualified and then having staff fit it to every substation, arrange the data links, and construct the dashboard.
I also wonder what the realtime requirement is. Data from a minute ago is fine .. except in this kind of situation, when things are changing very quickly.
Doing that at scale is tricky and requires a lot of people to participate in the mechanism, whereas the law only forces producers above a given size to participate.
The estimates we get from seasonal studies are usually close enough, especially since load shedding isn't a finesse exercise.
The situations that require load shedding usually give operators only a few minutes to react, where analyzing the issue and determining a course of action takes the lion's share. Once you're there, you want the actual action to be as simple as possible, not factor in many details.
They don't really disconnect it like that in these circumstances (as in they choose what to disconnect). As far as I know generation plants will start disconnect when grid frequency is <49Hz or >51Hz (at least in the UK) automatically as it's all starting to go very wrong. This is what causes this huge cascade effect. Roughly speak less frequency means there is more demand than supply and the other way round for higher frequencies.
This has changed a lot though, as even home batteries afaik will start discharging if they start noticing the frequency dropping to provide some support on generation. But if it's dropping too fast and too quickly it won't help.
But yes they do have very granular info on all the HV sources and how much load is on them.
Load shedding is an active measure taken when the grid operator knows that it will not be able to balance supply and demand. This is what happens in South Africa, where the operator preventively disconnects parts of the network at predetermined times to ensure network balance. As this is programmed, it is possible to rotate the areas under blackout.
In this case, we are dealing with a widespread grid incident. The various grid protection mechanisms have been triggered to prevent interconnection overload. In addition, the generators are trying to correct the grid frequency to exactly 50Hz. At 49Hz, more power must be generated; at 51Hz, less power must be generated. However, if the frequency varies too much, there are also protection mechanisms to prevent the turbines from overspeeding or amplifying frequency variations.
The grid is complex, and normally this type of incident is limited to one cell of the electricity distribution grid. A blackout is a domino effect, when a minor event triggers a chain reaction that disconnects more and more elements from the grid.
Th grid operator will have to restart or reconnect the power plants one by one, restore power to stations and sub stations. All of this must be done in a specific order before power can be restored to consumers. All of this takes time, requires resources (you need men on the ground), and the slightest error can lead to further outages.
Some consumers are prioritised, such as hospitals, transport infrastructure, telecoms and water networks.
Many critical pieces of equipment have UPS systems, but these are not always designed for such long outages or have not been tested for years. There are patients with home equipment who will struggle.
This is why rotating load shedding is preferable. The outages are not too long and vital infrastructure is not affected (or less so).
In my area, "buckets" of average load are prepared ahead of time in coordination between the transporter (who gives the order to shed), the distributer (who implements the shedding operation as close to consumers as possible), and representatives of the state (who define priority areas that can't be load-shed, or that should be load-shed last).
When yhe time comes, you just shed enough "buckets" to stabilize. Load shedding is not a precise task, when you're at that point, you'd rather load a few more megawatt and be safe, than play with the limits and be sorry.
> So how do grid operators know what amount of load will be cut if they disconnect point A123
Opening a line within the grid isn't used to shed load, as the grid is mainly redundant. It's used 1) to protect the line itself (either by letting it trip, or by opening it preventively), or 2) to force power to flow differently through the network, by modifying its impedance. Opening a line from the transport network is not a way to load shed.
In this totally random example [1], opening the line A-B increases impedance in the right part of the network, forcing power to re-route through the left part, and reduces the load on another line in the right path that was overloaded.
> Or is there continual measurement? (e.g. there's XYZ MW of load behind A123 as of 2:36pm)
The network is, indeed, monitored continually, and we do have potential network equipments failure simulations every few minutes, with contingency plans also simulated to make sure that they work in that particular case. This way, when shit hits the fan, we at least have a recently tested plan to start from.
I was going to say something similar. I live in Portugal and I've heard a lot of panic/fear mongering, mainly from the techies in the co-working space I was working on and expats.
(apologies for singling out these specific groups of people - my point is that it might be worth to put down news sources like xitter, and read AP/translated local Portuguese news)
I experienced it first hand in Madrid. This was much scarier than I would have imagined.
News travelled extremely slow: phone coverage was just barely enough to receive a couple text messages every 15 minutes or so. News spread on the street, I even saw a group of 20 people hunched around someone owning a hand-held radio in the streets.
Just before power was restored, things started to get worse, as the phone coverage went completely out (presumably batteries were depleted). People were in between enjoying the work-free day, and starting to worry about how tomorrow would look like if power didn't come back.
When the northeast blackout hit in 2003 in NYC I dont remember any panic. We still had house phones and they still worked in the blackout thanks to telcos being legally obligated to give a shit about reliability.
I stopped by a friends house and we then went on a walk. Some stores were open and cash was accepted. We hung out later that night and had a few beers. The sky was amazing as there was next to no light pollution. Next day was totally in the dark as well and again, no panic. More beers were enjoyed.
The choice to move to electronic everything without having to give a shit about reliability is a failure of modern government. Move fast and break society for a dollar.
> thanks to telcos being legally obligated to give a shit about reliability.
Yeah, they don't need to do that anymore. Around me, enough towers have battery backups that I can count on 2 hours of coverage when utility power goes out (if it goes out late at night or early morning, there's usually coverage until 6-7 am when people start waking up and use up the rest of the power). I don't have a real landline, but the telco DSL would drop instantly with utility power so I don't have big hopes and I wasn't willing to pay $60/month to find out.
Around when I moved, stores would pull out the credit card imprint machines, but those don't work anymore because cards are flat. Cash might work, and I've got some, but I don't think many people in my community do; people don't have cash for the snack shack we run at my kid's sports, so I doubt they have it for restaurants and stores either. And we get frequent 2-4 hour power outages, at least one, usually two or three per year; and ~ 24 hour outages every few years. The snack shack runs during summer where electricity is most reliable, but I doubt people stock up on cash in the fall and use it all up before spring/summer; they probably just don't have any.
> stores would pull out the credit card imprint machines, but those don't work anymore because cards are flat
It's the other way around: Cards are flat because a carbon imprint doesn't afford the merchant any payment guarantee by the card issuer anymore anyway. (In other words, the "floor limit" above which cards require electronic authorization is now zero.)
At that time merchants probably still checked the signature on the back of your card against your signature on a bill. These days nobody bat an eye when I use my unsigned card. I guess at that time a matching signature would give the merchant enough confidence to process the card offline.
The merchant's confidence is irrelevant if it's not backed by a guarantee of the scheme (effectively forcing the issuer) to pay even in case of fraud.
The people operating these imprinters are sales clerks and waitstaff, not graphologists or experts in detecting altered physical credit cards. The sophistication of fraudsters has also advanced, and as a result, a system that might have been good enough in a pinch 20+ years ago isn't necessarily good enough today.
That said, in my view there's no excuse to not leverage the physical chip present on effectively all credit and debit cards these days, which is technically capable of making limited autonomous spending decisions even with both the issuer and terminal offline in scenarios like this. It probably won't happen without regulatory pressure, though.
I started signing my cards with an all caps "ASK FOR PHOTO ID" 30 years ago. It raised a few questions when I would travel to the US and use them there, but was never refused a transaction.
That’s an urban legend, and stores are not required to actually do so. (And as far as I know, a thief could sign the receipt in all caps, ASK FOR PHOTO ID, and it would be a valid signature :)
In fact, even verifying the signature is no longer required in at least the US.
Signature verification also only solves cardholder authentication, not card authorization (i.e. figuring out if the card is funded, still valid etc.)
Please point at particular people and parties without wording it as "some politicians".
In Poland alone we have far right, a lot of centric parties and some more or less leftist parties.
The most popular (by popular votes) "right wing" PiS is not "right" for the most part for several years now, had a Marxist/socialist prime minister and gave so many social benefits away to grab votes that it made "left" blush. They are "right" only on the level needed to get church-goers votes.
And, with antisemitist agenta, PiS was buying Pegasus subscriptions from NSO to spy on opposition and unvafourable journalists just in hopes that they'll get something from it.
So - which "particular politicians" are you quoting?
2003 was over two decades ago. Most people didn’t have mobile phones let alone smart phones. I’ve been in Lisbon today and it’s surreal being 100% cut off from friends and family back home and a big relief power is back on, we’ve become very used to and reliant on seamless instant connection. Our mindsets and how we live our lives with that instant communication is totally different to 22 years ago.
> 2003 was over two decades ago. Most people didn’t have mobile phones let alone smart phones.
Maybe it depends on the country, but my memory of 2003 is almost every non-elderly adult I knew (in my own upper-middle class milieu) already had a mobile phone. Not a smart phone as we understand the term today, but a lot of phones back then had primitive smarts that are now largely forgotten, such as WAP/WML browsers (which maybe not many people used, but I certainly remember using one), JavaME applets (vaguely remember using them too-maybe post-2003, but higher end 2003 phones definitely could run them), vendor-specific mobile app formats such as Symbian
It must be. Most people I know got their first cell phone in 1997. That year over a few months it went from almost noone has a cell phone, to almost everyone having one.
That sounds like the sensible reaction, at the time at least.
It's interesting to think about and realize how much things have changed now though, and how reliant people are on everything, and especially their tiktoks etc. working all the time.
Some of the panic is likely related to the war in Europe too, and especially the general talk about war
Back in the 90's I worked at Nortel and visited a modest size Captive Office in Los Angeles. It supported maybe 20k or 30k people. I was amazed by the field of lead-acid batteries, 1.5m high x 50m^2.
Like 10-15 years ago before VOIP I sold internet/landline services door to door for a summer. My biggest selling point was explaining to people (usually who had children) that the VOIP service they switched to would not work in an emergency where power wasn't available... Worked like a charm to get people to switch back.
Similar experience in a town in Madrid's metropolitan area.
Electricty went down (something kind of frequent). My UPS kept PC up, and alarm system with sim and small UPS mantained wifi up for an hour or so.
Scary moments started when people I was in a call with in Portugal texted 'Grid is out'. Later no phone signal nor data.
At first, it might seem people running towards supermarkets an overreaction on being without TikTok for a couple of hours, but you have to live how scary it is to experience this in Europe's current political status to know 60 million people (plus industry) in three countries are out of the grid.
If you see Snowden's film (this might not be the most trustworthy source) it is exactly how CIA's agent describes the feasable attack towards these countries. Again, not a valid source, but I'd love to understand if that could be feasable.
Experience from Portugal, near Lisbon: fake news and made-up stories traveled fast! My wife called me (before phones went out) saying someone heard on the radio that Portugal was on red alert, it was WW3 (world war 3) and I think she even mentioned "missiles"! Also someone said it was a cyberattack and all Europe was off. Lots of panic reactions, many people buying toilet paper, water, candles, sausages and other canned food.
All gas stations closed because they could not sell gasoline/diesel. Today there are lines on all gas stations, people filling their car tanks and bottles..
Oh, let me tell you about electric cars! Many people had to spend the night somewhere away from home because they could not charge their cars.. My sister (with her job's electric car) had to stay the night some 200km away from home, and since the ATMs (Multibanco) didn't work, she didn't have physical money to pay for food. Luckily a stranger paid for the food (yogurt and some cookies).
Petrol cars, because of their range, had better luck!
Pure fear and panic..
I can only blame the authorities (Portuguese/European) for not having contingency plans for keeping people informed, and thus letting fear spread like wildfire.
Because you're always buying toilet paper in bulk, and everyone needs it.
Toilet paper is cheap and bulky, so stores only stock what they absolutely need. If a store is supplied once a day, it'll have something like two day's worth of toilet paper on its shelves.
Some incident happens, and people start to panic. By sheer coincidence one brand of toilet paper happens to be out of stock. The shelf space is huge, so as you walk past it you immediately notice it and think "geez, wouldn't it be awkward if that were to run out". You don't know how much you've got remaining at home so you grab a pack, just to be sure. Ten people do this, and because the packs are so massive another brand has just run out - making even more people consider picking up toilet paper.
You normally buy toilet paper maybe once every month, so if only 1 in 15 people pick up toilet paper during their panic shopping, those two day's worth of stock will quickly run out!
I've asked the same question. I think people are so pampered in western societies that the worst thing they can possibly imagine is not being able to wipe their ass with a fresh piece of toilet paper.
If you are old enough to have a conversation with someone who lived through the 1930s United States Great Depression, they had to resort to tearing pages out of the Sears catalog. After that, they would hoard toilet paper. And every generation since has done the same thing. But it started with the Great Depression.
During a bad storm here many people used their electric cars to power their house. There were still places to charge it and then come back to the house to power it until the house was re-connected.
I think this is where F-droid and Briar have a (short-lived in this instance) chance to shine. Since Briar allows communication between phones without access to the Internet, and F-droid allows to direct transmit apps between phones as well.
I wonder what similar solutions exist in the iOS ecosystem.
we tried briar during the recent mass protests in Serbia and the bluetooth connection never worked, it was unable to connect to any of the contacts that were previosly added using the normal 4G connection.
My son was at the university outside of Barcelona, and I lost contact with him for 6 hours. He was traveling home, and I knew that the roads had no working semaphores, and there were dozens of incidents...
I lost some years today.
My son is fine, thanks to a random person (“the man with a rabbit”) who just decided to give a lift to my son and his friend to the edge of Barcelona.
I come from a world where I faced power outage for 17 hours for weeks. Each day there would be power at random time of the day. Life is just bleak. And when I see these kind of posts, I sympathize with them. You can't do anything in that situation. Modern world itself is so complex, you are left to thoughts. If this continues for a month, most people will go insanse especially those who rely on technology to stay sane.
no one has emergency fm radios there? I thought that was pretty much ubiquitous. that's the first I'd go to check what was going on I think, other than my phone
From my personal experience, mostly just in the car. Here in Portugal there was a rather well-timed rail strike today, so a lot of people who would've taken the train drove instead and had a way to get back and listen to the radio. From what I could tell from the radio reporting, handheld radios and batteries were rather quickly sold out.
I've got to give massive props to Antena 1 too, which is the national broadcaster's main radio station, who stopped all normal programming and did an all-day massive report on this situation to keep people informed. From what I could tell they didn't even run any ads during that period, just all-day reporting continuously repeating key information for people who'd just tuned in.
> From what I could tell from the radio reporting, handheld radios and batteries were rather quickly sold out.
Indeed, visited the local hardware store and some bigger stores like Decathlon, and they were out of all batteries, gas-powered stoves and anything else related. Seems they ran out of it just hours after the power was cut too.
> I've got to give massive props to Antena 1 too
RNE did a great job, and together with the response of the previous crisis of Covid, I feel relatively safe as a Spain resident during a crisis. People around you are so caring as well, like when I tried to figure out how to open my garage door without power, a young guy stopped while passing by to ask if I needed help. Simple stuff, but gives a larger feeling of that we'll survive together no matter what.
I did not see much fear. I was at work and it took about two hours for us to realize the outage was not just local. The cafeteria had gas burners and served everything they could to empty the refrigerators. We all at lunch and discussed whether those who lived far away (train trip) would need to sleep at work (they might have, I don't know what happened to them). I made the relatively short 75 minute walk home across the city.
The atmosphere was quasi-festive and most people were quite relaxed, enjoying an unexpected afternoon off. Younger people filled the bars which were serving everything they could. There were long lines at supermarkets and an occasional fellow toting a box of supplies, but mostly there were just huge numbers of people in the street and completely collapsed traffic flow (the police were out in force almost immediately, directing traffic). In the part of Madrid I was in about 1/4-1/3 of the population is from South America and I suspect most of them have seen this all before anyway. The only real stress I saw was from people that need a train to get home (because the trains weren't running) and a had a walk of more than 2-3 hours.
I got cell phone signal when I was near two hospitals which were fully operational.
It was interesting that almost immediately, while I was still at work, everyone said power was out in Portugal and France too. After an hour or two some were claiming problems in Germany, but this seemed already to be unfounded rumors.
Some younger people couldn't walk home because they didn't have google maps ...
What caused it?
The Portuguese prime minister, Luís Montenegro, said that the issue originated in Spain. Portugal’s REN said a “rare atmospheric phenomenon” had caused a severe imbalance in temperatures that led to the widespread shutdowns.
REN said: “Due to extreme temperature variations in the interior of Spain, there were anomalous oscillations in the very high voltage lines (400 kV), a phenomenon known as ‘induced atmospheric vibration’. These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European network.”
They have data and probable causes, but nobody will tell until forensic studies take place, which seems a good decission. No need to spread speculations.
A cold start from zero generation took 6 to 10 h. No casualties, hospitals and crtitical infrastructure kept working all day.
Whatever the cause, this has been a great test of the emergency protocols.
I'm seeing some reports saying that a significant frequency oscillation happened, which triggered automatic shut-downs, which cascaded. Could an event like this have that effect?
I suppose it makes sense that it was an automatic shutdown rather than infrastructure failing on such a wide area. And then once it's shut down, a black-start is a logistical challenge as other comments have explained.
I'm also seeing some reports about it being more likely that something happened on the east side, somewhere like the Ebro valley or north across the Pyrenees. Catalonia seems to have been particularly affected, and it's on the path of important lines coming from France. High heat at noon could have caused a line to fail and short against a tree, which would be similar to the 2003 nation-wide outage in Italy.
The grid is supposed to tolerate any single failure, even under full load. Of course sometimes the first failure is a fire or equipment malfunction and the second failure is a planning failure or someone pressing the wrong button.
Cascade failures are common. The weakest link fails, load gets re-distributed evenly, the second-weakest link fails and so on.
In theory [a flawed one] you've had enough spare capacity to survive N failures and N+1 failures are statistically unlikely because p^(N+1) is close to zero.
On practice [or with a better theory] you can't multiply probabilities in a grid system because random variables aren't independent. 30% spare capacity can go to -100% in a second.
Grey failures are harder for large systems to handle. If a chunk goes hard down that's usually easy. Something like voltage oscillations that trigger cascading failures in a sequence can lead to negative feedback loops that bring it all down.
From Le Monde live feed, RTE (French electricity network manager) declared the issue unrelated to this fire.
"Le gestionnaire français souligne par ailleurs que cette panne n’est pas due à un incendie dans le sud de la France, entre Narbonne et Perpignan, contrairement à des informations qui circulent."
The 2003 US Northeast blackout was caused by the failure of only a few lines that shorted into trees. These line failures created grid instability that resulted, ~5 minutes later, in most of the Northeast losing power in a cascading failure.
Redundant systems have failures in one path quite often that you never know anything about. We get headlines when the failures correlate in the same timeslot.
Few years ago nearly entire day European network was sitting on N-0 due to multiple issues in Poland, caused by a heat wave and deeper root causes. There are many power plants and power lines where any further issue would cause Europe-wide blackout.
you are right, but the emphasis could use a tune-up. In California, home of world-leading tech.. there are sensors and information networks, extensive electrical power lines, heavy equipment and budgets, a lot of dry and dead tress, a history of fire. So you see that California in a way is a world-quality testing lab. and the way the information travels, and the way the information is applied, could also be world-quality .. or, world-theater for government imbecility..
It looks like the Iberian peninsula is relatively isolated from the rest of the CESA synchronous grid, with only 2% cross-border capacity compared to local generation. [1]
There's a map at [2]
> The Spanish electricity system is currently connected to the systems of France, Portugal, Andorra and Morocco. The exchange capacity of this interconnection is around 3 GW, which represents a low level of interconnection for the peninsula. The international interconnection level is calculated by comparing the electricity exchange capacity with other countries with the generation capacity or installed power.
That graph doesn't seem to make a very clear distinction between historical, real-time and predicted values... I think the event happened at 12:30 local time or so.
There seems to be some kind of recurrent daily pattern where the French - Spanish interconnect switches from Spain -> France imports to France -> Spain exports at around that time, and then back again in the late afternoon.
One guy simulated Australia's electrical grid for 3 years to show that the entire grid could run off of >98% renewable energy using the installed renewables with just a small amount of storage (e.g. 5 hours of average demand, ~10% of demand over the entire period).
To me this indicates that in many energy markets where renewables are sufficiently built out, the only factor for why we aren't using them more is the storage capacity and grid infrastructure to handle their variability -- and instead just running stable but dirty energy systems.
There are a lot of cool mechanical grid systems like gravity batteries (e.g. weights on a pulley system) [2] or compressed or liquid air storage systems [3] which provide cheaper storage at higher capacities and durations than electric batteries
It makes intuitive sense to me that a small amount of storage would make a big difference. Outside of peak use there has to be a huge amount of potential effect that never gets generated because lack of demand at that time.
Yeah, this matches what I've been thinking too - we're way closer to a high-renewables grid than people realize, but the bottleneck isn't generation anymore, it's storage and infrastructure
A similar thing happened last June, with Albania, Bosnia, Montenegro, and most of the Croatian coastline losing power simultaneously.
Definitely felt surreal to first lose power to the degree that even traffic lights were no longer working, and then to hear it's also happening across the region just before mobile networks also went offline.
It shows that even when the original issues are solved the reboot takes time due to power station being spooled down due to excessive production. I hope it is all back before telecoms start draining the batteries, otherwise things get uglier
Older readers may remeber the Northeastern blackout of 2003 in the US and Canada that was caused by cascading overloading, I believe originally triggered by high loads due to hot weather and poor vegetation maintenance under lines.
I was an an adjacent area at the time and iirc we were saved by our nuclear operator releasing some insane amount of steam to bring the supply down and avoid more overloading.
I have such great memories of the blackout. I was in college at the time, so obviously indestructible.
The first night, we had 2 grills with 10ft high flames roaring but I would say in a controlled fashion. The cops quickly asked us to not do something dumb like that anymore and we agreed.
Memories of the local party stores selling 40oz's for pennies on the dollar, as the outage would last days but the booze would not. It didn't really turn into anything bad for us in SE Michigan other than a fun story I can tell my internet pals about 20 years on.
If I recall correctly, we were also somehow the first house in the neighborhood to get power restored...I remember playing a Dreamcast with a hacked NES emulator running Rampart 2 player. I never made good on it, but I always said I wanted to make a shirt that read "I blacked out in the black out of '03"...which is probably for the best.
> In January 1998, Montreal and the region surrounding it was hit by the most disastrous ice storm ever recorded: more than four inches of ice entombed an area larger than the State of Florida, causing trees and power lines to collapse on an unprecedented scale, leaving millions in the dark without heat (some for up to four weeks). 35 people died and damages totalled more than $5 billion making it the worst natural disaster in Canadian history.
Portugal has no electricity as we speak. Funny enough telcos and 4G/5G are fine for now, I'm guessing batteries and diesel backups kicked in and are doing their job.
Update: still no electricity, 4G/5G is barely working, city is more chaotic than usual but not that dramatic as some media say - there are huge queues to the buses and smaller shops that are still working, more people are outside.
Old-school PSTN folks looked at XKCD 705 and chuckled. Late to the party, pal.
The telephone network was designed from the ground up to be completely independent of _everything_ except fuel deliveries. If grid power is up, that's convenient, but it's totally not required.
In many places, that's because telegraph and telephone lines got there before the power grid did. Lines running along railroads connected communities that had no centralized power generation. Delco-light plants at individual farms might be the only electric power for miles, aside from the communications lines themselves. Even if the only phone was at the rail depot, it still had to power itself somehow. As those communities sprouted their own telephone offices and subscriber lines branching throughout town, the office had its own batteries for primary power, and eventually generators to recharge them. (Telegraph networks largely ran from just batteries, recharged chemically rather than with generators, for years.)
Fast-forward a century and there was just never a need to depend on anything else. As long as the diesel bowser can get down the driveway, the office can run indefinitely.
Among old AT&T/Bell/WECo hands, the devotion to reliable service goes far beyond fanatical. Many offices built during the cold-war have showers in the basement and a room of shelf-stable food, though these are no longer maintained. The expectation was that whoever was in the office when the bombs dropped, would keep things running as long as they could. And when they couldn't anymore, well, there was probably nobody left to call anyway.
Depending on country there are sometimes very strict requirements - or just traditions sometimes - around building up strong survivability in face of total loss of grid power. Including diesel and turbine generators on bigger BTSes let alone exchanges. If you drop capacity per terminal (so bandwidth) you can cover a lot more range at times which helps with mobile network resiliency.
> Including diesel and turbine generators on bigger BTSes let alone exchanges.
Or if you’re AT&T, grid natural gas backup, so your CO goes down if electrical and natural gas both go out once the batteries die. Did I mention how they didn’t build in roll-up generator connection points and had to emergency install those?
Are you sure "offline" means that? Romania looks offline and when I checked their CNN they were reporting live from Spain about the blackout without mentioning Romania.
The map seems to be based on monitoring stations in the different locations, so yes - it's also possible that a station is offline for other reasons (maintenance, etc).
In any case, if I recall correctly from a Youtube video I can't find (it was either Wendover or Real Engineering), if the grid is fully down, it takes quite a lot of effort and time to bring it back online because it has to be done in small steps to avoid over/under loading/using.
Last big outage here in Spain, we got a generator, wind turbine, solar and battery. All cheap Chinese stuff as it doesn't happen often, as a result, this event was barely noticeable to us. Especially the wind turbine did well the entire day and night until the power came back a few hours ago.
We are prepared, just have the people in the suburbs breed some more and ship them over to the decimated cities. You can think of people in the cities as the rat-wheel of production. No culture is better than any other culture, therefore all culture is expendable.
No, not realistically. There are physical and economic constraints. There's no reason to over-prepare either. It's absurd to remain as catastrophically vulnerable as we are though.
Well, sex and looting, and maybe inventing new music genres.
The legend is that hip hop and sampling started after the blackout in NYC. Some electronic and music stores were looted and the recording equipment eventually ended up in the hands of musicians looking to make a new sound: https://www.rollingstone.com/music/music-features/new-york-c...
We had a (much smaller) regional (Lübeck and Ostholstein) power outage here in Northern Germany in I think 2018 or 19. No mobile internet until everything was restored ~6 hours later, and most of the time no other cell service either.
Always picture someone with a big can of diesel filling up a generator. Experience says it could be a genny running on natural gas. Had that at work before.
I think cell phone towers use surprisingly little energy, just a few kW. So even longer term operation should be possible withs bobs backyard generator running on vodka.
But I would guess the whole network equipment would draw quite a bit, especially a modern infrastructure.
I live in Madrid, and when the electricity came back around 22:00, people started cheering and celebrating in the street. You could hear how each neighbourhood was going back online.
It’s funny to think how the moment is goes off you feel nothing, but hearing that many people produce noise and express happiness makes your body notice instantly, a sensation we often describe as electric.
Grids are tricky, because the electric socket is kind of a leaky abstraction for heavy plant machinery in turbines of thermal plants, load balancing, synchronisation, and a gazillion other things I really don't understand.
Is a grid built on renewables and batteries somehow more resilient? Solid state things tend to be less fiddly, hence my question.
I remember reading at one point in the past that renewables were actually worse for the grid due to less predictable power generation or something, but that was a long time ago, certainly pre-battery storage.
I'll give a somewhat simpler answer than Filligree has. The problem with renewable energy sources is that they are typically both highly variable and not dispatchable (i.e., controllable). The former leads to supply peaks that can exceed transmission capacity or supply lows that require compensating generation elsewhere. The latter means that energy generation can easily be increased or decreased as required, which is of course very helpful for grid management. Dispatchable generation can be increased if supply requires it, or multiple dispatchable generators can be 'redispatched' to relieve congestion in a part of the power grid (by rebalancing generation). Power from renewable sources can be decreased through curtailment, but that wastes the generated energy.
The "problem" with renewables is that they are not dispatch-able, meaning you will never be 100% sure that tomorrow or a week for now there will be sun or wind in a specific region. On the other hand, a nuclear or a coal powered plant, know months in advance if they will be available for peak load or not.
Yes, if it is sunny or windy, they can be scaled in minutes, but only if conditions are met. The inverse is true for nuclear/coal - they cannot be scaled up & down in minutes.
Yeah, hence I mention batteries specifically in my question. I guess a renewable grid will need a lot of batteries. And presumably these are highly dispatchable?
> Is a grid built on renewables and batteries somehow more resilient?
So this is a complicated subject in itself, and a full answer won't fit inside this textbox. Some bullet points:
- Grid stability is maintained by batteries, but not literally. The "batteries" in question are typically rotating generators, i.e. turbines, wind, literally anything where you have an electrical coupling to a lot of physical inertia. That's what keeps the grid running second-to-second; while a power plant might pretend it's outputting a constant 4MW, it actually shifts noticeably from moment to moment. The kinetic energy of the generator helps balance that out.
- Going up from the sub-second range, an overload of the generator obviously would cause the shaft to slow down, dropping the frequency and causing brownouts. Brownouts are bad and can damage the grid, so typically breakers will disconnect if it falls below 49Hz; a 2% drop.
- Baseload plants can't cope with this, as they take multiple minutes to spool up. Minimum; for something like a coal power plant, where you have to shove in additional coal and wait for it to catch fire, it's going quite a few minutes. This is what defines 'baseload'.
- Peaker power plants can increase (or decrease) their mechanical power production in a matter of seconds. These days that typically means gas turbines, though hydroelectric power is even better, and nuclear power could be used for peaker plants -- but isn't; most nuclear reactor designs outside of the navy is a baseload design. France does have some load-following designs, and we need more of those.
- Wind turbines can't increase their output, flat out, but they can decrease it (by feathering, or by using brakes). This is good enough, except this would turn them into 'peaker plants' that can't help with peaks. If we had enough wind turbines to cover 100% of the load then we'd technically be fine, but economically speaking that doesn't work; they'd be at less than 10% power most of the time.
- Wind turbines have rotating shafts, but a lot of the time they produce DC power, linked through inverters, which removes that benefit and makes them act like solar panels in effect. However, this is a purely economic issue; they can trivially be upgraded to support grid stability if the pricing scheme will pay for it.
- Solar panels are worse: They have no inertia! There is no rotating shaft there to cover sub-second usage spikes. That's where complaints about 'renewables causing reduction of grid stability' come from, along with issues like domestic solar needing to backfeed power through distribution lines and transformers that aren't necessarily designed for that.
- But batteries can absolutely help. The kinetic energy of a rotating turbine isn't actually that big; it's not that expensive to pair a solar panel with a battery to build a grid-forming system that acts the same way a kinetic power plant would.
For wind turbines, solar, and batteries, the frequency stability is doable through controlling frequency from inverters.
Some wind turbines are also internally a hybrid design that can dynamically adjust the frequency difference angle both for minimal losses in production, but also to provide frequency shifting and even artificial demand (i.e. essentially using wind power as brake)
The inertia provided by synchronous generators was in essence a natural benefit to the grid that we never appreciated previously. But now with more renewables it’s now something we have to consider and design for. Running a grid on 100% non synchronous requires a bit more effort!
It almost sounds like running a large motor/generator with a large flywheel on it, just for the purposes of frequency stabilization and flattening load spikes, may be a good idea in a large solar installation. The downside is the mass and the bulk, and the cost of copper. The upside is zero reaction time.
Possibly some kind of rare cascade failure where it might have been on the edge for a while, and some small event happened that tripped things, similar to the american northeast blackout in 2003. High demand, plus a power station going offline meant more demand on some interconnects, which shorted on trees and were cut off, putting more load on other lines until the entire system collapsed
Amazing/curious that all the infrastructure to feed news to your phone still works. I can imagine a poorly-built systems would have some infrastructure without UPSes, and in all that chain of technology, there'll be an element or more without power. Not saying Spain/Portugal would necessarily be more prone to have poorly-built infrastructure, I assume the whole world runs on half-assery.
If their networks are built like the ones I've worked on, everything has at least a little battery, but only some of it has generators to keep going after that.
So some or most cellular towers will have generators, and their fibers will backhaul through repeaters, some or most of which will have their own generators. When it gets back to the MTSO, that will definitely have large diesel turbines on site and at least 24 hours of fuel with priority refueling contracts.
I'd expect there to be a lot of outages, for instance where all the towers in a region end up backhauled through a site where the generator fails or was never installed for some reason. But there will also be a lot of places that stay up in some capacity because, more or less by happenstance, all the fuel tank permits got approved and all the equipment actually worked.
The European grid is a connected synchronous grid. Usually that would add stability but it also means if a single country's grid blacks out then their neighbours have to respond to that. Portugal and Spain's grids will be intimately connected which won't have helped.
The whole European grid is essentially connected at this point, it's a simple trade of resiliency for efficiency. People just forget about the resiliency part until something goes wrong
For that incident, an expert panel was set up in July, the interim report was published in November, and the final report in Feburary 2025: so it'll take a few months.
Not required, but engineers tend to enjoy this sort of thing. Also, since it affected some 60 million people and EU-wide grid interconnects, someone will have to explain what happened.
> A fire in the south-west of France, on the Alaric mountain, which damaged a high-voltage power line between Perpignan and eastern Narbonne, has also been identified as a possible cause.
To remember: in the early hours of this morning there was a huge purchase/transfer of monero that increased the price +30% (highly unusual) and a few hours later before lunch time we see Iberia disconnected from the grid.
Monero is the favorite payment coin when ordering real cyber attacks.
yes (of course) .. It's fascinating how always people love to jump to conspiration theories .. If it were a cyberattack (and so far it isn't proven at all), it would be nothing less than a state sponsored cyber attack to be able to get down the electrical grid from 2 EU countries - and obviously no "reward" such as crypto would be involved behind the scene
Please refrain to call it a conspiracy when it is a valid hypothesis with valid facts.
It is common to use Monero as reward for specific groups performing cyberattacks. Please inform yourself on the topics and reflect afterwards on what was written.
Please read carefully what you linked. It claims theft without a victim being identified. There is no victim and there is zero evidence of being stolen coins.
yes really relevant/related, as that talk also mentions that a sub-1 Hz variation of the nominal 50 Hz is already enough to cause blackouts - which is also the case for the current blackout: https://x.com/Nexuist/status/191687508022891747
the talk discusses further aspects of the European energy grid which are relevant for the current situation.
> The used radio protocols Versacom and Semagyr, which carry time and control signals, are partially proprietary but completely unencrypted and unauthenticated, leaving the door open for abuse.
You don't think the omission of encryption and lack of authentication imply a criminal absence of due diligence.
I guess some huge contracts are in the offing fix those necessary features which should have been built-n in the first instance.
Because to buy that much monero so fast is only possible with bitcoin, which is traceable by governments and interested parties.
An estimated 305 million USD of monero were purchased this morning before the grid shutdown. That was a +50% increase on the monero price, something that never happened before in the last 11 years of the coin.
I am currently in my work office in Madrid, main building has electricity so I guess they have some backup generators, the kitchen however is out of service.
According to local newspapers metro network, airport and traffic lights are all down
Dunno about Spain/portugal, but in North America they’re usually just for elevators/alarms, partial hallway lighting & maybe water pumps (for firefighting).
Maybe a tech company with servers will pay extra for full backup, but it’s not typical.
Without fail, during every grid outage, some will fail to start and there will be elevator rescue calls throughout the city.
To use a potentially controversial example, Microsoft products (Office, Windows) are still extremely entrenched despite the overwhelming majority of knowledgable people agreeing that they're on a steep downward trajectory and the alternatives have long since surpassed them.. leading to this[0] recent video from Pewdiepie...
I don't know why you are so enthusiastic about this site. Whatever, my assumption was that they are displaying the data in 1 hour intervals as the picker suggests and showing a warning currently about the affected regions is consistent with that assumption.
I saw it on twitter and had to go to BBC to make sure this isn't some joke or hoax. 2025 and we have National level power cuts?
I know it may be rare but I think some day we really need to move or mandate every single flat / home / apartment / living places to have a 12 - 24 hours backup battery included. Something that has 10K+ Cycles, durable and non-flammable. Not only does it make sure our modern lives without sudden interruption, it also solves the renewable energy problem.
Most of the time, the grid is reliable. How much investment do we actually want to prevent a few hours of downtime once in a blue moon?
And even if we do want to invest in large amount of grid storage (which we would need to anyway, if we want to transition to renewables), I'm not sure pushing this down to the individual house is sensible. Its a great way to limit economies of scale and make maintenance/inspection harder.
Mandating battery storage is nonsense. Making the regulatory system better for it is not, though; it would be good to make it possible for consumer battery systems to do "islanding" properly so they can be true backups.
Or at least not making it illegal. Regulate it all you want, make sure it's safe, but please let's let homeowners install a transfer switch if they're willing to pay for it.
In a longer outage it'd be a massive benefit to have even just a few houses here and there that retain power.
> I'm not sure pushing this down to the individual house is sensible.
why not? Having distributed buffers ought to make the system more resilient in most cases wont it? Not to mention that these home level batteries can be used to smooth out power usage and lower peak loads.
In fact, having an EV car act as this same battery would be an even more efficient use of resources.
Why not? Maybe because I'd rather not have to spend that much of my own capital on something I'd generally never use. I'd rather not spend the maintenance on that thing I'd generally never use. And finally, I'd rather not take up the space in my home with a large thing I'd generally never use.
I'd prefer for the power coming to my house to just not go out. The grid operator can install batteries in places other than my home. The grid operator can maintain them for me. The grid operator can get cheaper loans than I can for installing them, they're staffed with supposed experts in this stuff, just have them handle it on their premises.
Batteries can be useful for more than just getting through power outages. More and more utilities are offering time of use based pricing. For people on such plans a battery can be useful even if there are never any power outages by letting you store power from off peak times to use during peak times.
Unless it's a really massive swing it'll be a gamble if it will actually break even. A useful battery install for a home is at least several thousand dollars. I'll install $10k worth of batteries and manage to arbitrage $30/mo off my electrical plan, excellent! I'll break even in 28 years, wonderful! Will the batteries still have a meaningful charge after 28 years of use? Who knows! Meanwhile I'll miss out on the opportunity cost of that $10k over that same period.
And once again, another point for growing the wealth gap. Poor people who can't afford the thousands of dollars for installing the batteries get shafted by such things instead of us assuming the grid operators will just smooth these prices out for us.
Once again in the end I'd just prefer if there were actors on the grid running massive batteries able to arbitrage the extreme spot prices and sell me electricity at a reasonable rate all the time instead of me having to actively min/max every dang thing every moment in my life.
Doing some math on that $10k for an example of lost opportunity costs, today you can get a 10yr CD at 5.5% APY. Over a decade that's a hair over $17k, practically guaranteed return. So, to properly break even you're going to need to not just save $10k worth of electricity, you'll need to save something more like $17k. You'll need to arbitrage at least $141.67/mo on average for that entire 10-year period to break even compared to what you could have had just investing that into pretty safe savings.
Personally, I'd rather toss that $10k into my kids' education funds and sign up for a fixed rate electricity plan. Hopefully that'll be better ROI.
Grid-scale batteries have pretty different economics than some batteries in my garage. Their real-estate cost is probably significantly lower than the cost for halfway decent residential. They're buying a much larger order of batteries, so their per-kWh cost is probably a good bit lower. Also, their installation cost per-kWh is also a good bit cheaper. They're probably completely fine buying/selling purely on open spot markets, meanwhile in the end I'm going to need to run my pool pump and I'm going to want to do dishes and laundry and charge my car and all the other things on some reasonable schedule so I'd like some amount of price protection on whatever time-of-use plan I get (a day of $9/kWh electricity prices will surely wipe out whatever gains you might have made). They're probably able to get cheaper loans than me, so it's easier for them to arrange the big capital investment instead of high interest loans or having to invest existing capital.
I'm not arguing the economics of batteries don't make sense in today's power markets. Far from it, I've been toying with getting into it in North Texas. But buying a few kWh of batteries and putting them in my garage probably isn't going to break even and almost certainly isn't going to make me money.
Further thinking about what you'll actually get, it looks like a decent average price per kWh of installed home battery backup is about $1,000/kWh. So if we got $10k of batteries installed, you're looking at 10kWh of energy you can store.
Let's say you're in California, which has a pretty decent swing in energy prices available to the home consumer. You'll see prices swing between $0.27 to $0.65/kWh. You've got a 10kWh battery pack, so you can arbitrage something like $0.38/kWh * 10 kWh = $3.8/day, assuming you get a full charge cycle and ignoring efficiency for simplicity's sake. $114/mo. So assuming you didn't have any loans and you're ignoring opportunity costs on that $10k, your break-even is in ~88 months or ~7.3 years.
After a decade of ~$114/mo, you'll have offset $13,680, assuming you always used a full charge and you experienced no other maintenance costs and bought it cash on hand and that energy prices didn't change and had a perfectly efficient inverter to charge and discharge the batteries and the batteries still had 100% charge capacity the whole time. The 10-year CD, which you could just forget about for a decade, stands at $17,310.
And this is for one of the few markets in the US that really offers that big of a time of use plan other than "free nights and weekends***" (with pages and pages of fine print and other fees) For instance, I just looked at a time of use plan available to me here in North Texas. 9AM-4PM I'm billed at $0.068 for the generation, $0.050029/kWh plus a flat $4.23/mo for delivery. Outside of that window I only pay the delivery cost. But that means I'm only really able to arbitrage $0.68/day with a 10kWh setup. That's 490 months or a hair over forty years to break even.
The more batteries available for that arbitrage, the less effective it is. As a grid user, I'd rather the grid own and operate those batteries. Certainly some of my outages are due to line issues between me and the substation, but I think more of them are due to line issues between the substation and its upstream. Having storage at the substation could potentially help all the customers and centralize the maintenance and safety burdens. They could do some energy cost arbitrage (although I live in the PNW and we generally don't have time of use cost distinctions, because hydro doesn't care)
I do have a 35 kWh propane fired generator onsite though, which does provide for outages regardless of where the break in the line is.
Sure. Having large batteries is an additional fire risk in my home for once again something I'd generally never need. If these batteries are in my home but managed by the grid operator I'd probably have to allow their people into my home to service them which I'd rather not have to do if it's something I'd generally never use.
Also, this further just makes having stable electricity yet another thing in the wealth gap. Only those wealthy enough to afford the high upfront capital costs, the ongoing maintenance cost, and the space to store it get reliable electricity, fuck everyone else! Or we can just focus on investing in a stable and clean grid and share that cost with everyone, all you need is to just be connected.
But hey if I get a massive battery bank I'll have power for when the end times come. I won't be able to go get groceries anymore and eventually the fiber line and radios around me will go quiet but at least I'll be able to play videogames. For a few hours at least.
Don't get me wrong, I'm an Eagle Scout, be prepared and all that. I've got a big pile of charcoal, a chunk of propane, a camping stove, several day's supply of water and canned/non-perishable foods, some batteries for lanterns, etc. The cars all get topped off when big storms seem possible. If the big outage comes this will be more worthwhile than being able to turn on the TV for a few hours, and cost significantly less than several dozen kWh of batteries. And if the power is out for more than a week or two I'll have far bigger concerns than being able to post on Hacker News.
You're not wrong about the fire risk, but just to note, house batteries nowadays are almost always LiFePO4 -- which don't catch fire the same way LiPo batteries like to do.
They can still do so for other reasons, like a short circuit in the wiring.
Fully understood and agreed. That's why I bothered saving that for the second comment, because it's not necessarily that high of a fire risk but it is still a little bit of one. Having that much energy stored in that small of a package will always have some kind of risk of "what happens when that stored energy gets released in an uncontrolled and rapid manner?"
And that said, I do have lots of other somewhat beefy batteries around the house. They do a lot of useful things for me such as power my tools including my lawn mower, string trimmer, hedge trimmer, saws, etc. There is a massive one in the car parked in the garage. In these cases it is a useful trade off of that slight risk as I'm actually getting something normally useful out of it.
The peak load of a house is substantially higher than the per-house peak load of a city. People don't all turn on their ovens/dryers/hairdriers at the same time.
You could distribute this capacity at each house and feed it back to the grid during peak times. But is TCO of 1000 * 100kWh same price as 100MWh worth of capacity?
If you're going to have the battery anyway(a car) its hard to compete with, but once you need more capacity, I'm not sure it makes sense to distribute it quite as much.
With places that have time-of-day rates, I wonder what will happen when enough systems get advanced enough to precisely kick-off/on (or slow down/speed up) in unison at precise times.
I’ve set this up on my “smart” thermostat to speed up just before rates go up and kick off for a while once they do (and it was a pain to setup, somehow this functionality was not baked in)
How many people will turn off AC and will not bake that cake, or wash clothes, when they know they're going off their own battery vs some electricity distributor's battery shared by others?
And if someone is dumb enough to do high load stuff on their own battery during a blackout, then it's less of a problem for others. Also individual failures will cover less homes.
Incentives and consequences will be different and differently spread.
And on individual level, you can also chose whether you want this or are fine with outage. (I'm against mandating this)
I’ve read discussions about preparing for a strong solar flare basically simply involves having on hand spare equipment stored unconnected and that then would also immediately improve repairs and maintenance for regular issues. I don’t know the details but I believe the US is looking at providing funding to the utilities companies to acquire the duplicate extra equipment.
Didn't we used to pump steam around for heating and now homes have individual furnaces? If the cost for this were about the cost of a furnace I don't see why this wouldn't be viable.
That's a great example of why this is a bad idea. District heating is a lot more efficient than using individual heat sources. Especially so when the heat being used is heat that would otherwise go to waste (e.g. waste heat from industrial processes).
I think we can have both? I don't think anyone is advocating for one giant battery. In my ideal imaginary world, there are medium sized battery parks near every transformer station. In that same fantasy, the foundations of wind turbines are also batteries, so I shouldn't pretend that this is hard science.
First-gen district heating was steam, then moved to hot water. It didn't go away [1] but there's regional differences. In cold climates there are oil based simpler backup plants that can cover for the normal CHP[2] and elctric+heat pump based generation. It's a good way to buffer and use cheap energy in periods of cheap electricty from eg high wind production[3].
In Polish cities there are attempts to move in exact opposite way (close individual furnaces) due to rampant pollution during winter (1000% of PM 2.5 norm level of pollution happening occasionally, over 400% PM2.5 norm lasting for weeks etc)
People seem to love those expensive solutions though.
I know so many people who invested in overly expensive battery storage systems for their solar. Power in Germany is expensive, but even with that expensive power many of those battery systems will never hit a positive ROI. But they’re still happy for the feeling of being „independent“.
I’ll turn 26 in a few months. The first time I experienced a power outage in my life was two weeks ago when a construction worker in our basement drilled into the wrong wall…
It's likely that battery backup solutions won't be expensive for long. Battery prices have been falling exponentially similar to Solar, and there is no reason to think it will slow down anytime soon (there is still room for improvement just from economies of scale).
I predict that home batteries will become a "no brainer" from a financial perspective (for anyone who has the upfront capital to purchase them) within the next 10 years.
I haven't seen battery price quotes for retail customers for a while now, everybody I know who wants one has one. But not too long ago a battery bank including installation was easily in the range of 800 to 1000 EUR/kWh of storage.
Battery storage may become a bit of a no brainer from a purchasing price point of view, but I don't think it will actually be more beneficial for people, especially if BEV adaption continues at a similar rate and bi-directional charing becomes widespread
You can put together a simple and fully independent solar island with 5kw/h capacity and 7kw nameplate generation for $4100. This is a plug and play system, with zero wiring and no knowledge needed. For the idea of the kind of profit margin that includes for the company, their "smart isolation switch" is $1k. Jackery has several competitors, and all of their build quality is pretty good, and the warranties are acceptable.
20kw/hr of just storage is about $10k. Every one of these companies also has standard DC input ports for bring your own solar panels, because their panels are overpriced.
The 8-Bit guy has been using such a system to maintain his studio. It is reliable, uses cheap solar panels. This stuff is commodity.
Not all of us live in (semi-)detached homes, and I’m also not sure how one would implement this at the building-of-appartments level, as in, who’s going to pay for it and how do you handle the battery storage capacity for hundreds of apartments from the same building in a safe and cheap-enough manner?
I was living in Brooklyn, NY during the 2003 blackout that knocked out much of the northeastern US for a couple of days. In those days we weren't quite so dependent on cell phones and such things. People walked home, since the trains were stopped. Guys grabbed flashlights and directed traffic. We listened to the news on the car radio. After that I went to J&R and bought a wind-up emergency multi-band radio. In the intervening 22 years, I've never used it. Though at the rate things are going, maybe that will change.
- "[Portugese electric network Redes Energéticas Nacionais (REN)] said: “Due to extreme temperature variations in the interior of Spain, there were anomalous oscillations in the very high voltage lines (400 kV), a phenomenon known as ‘induced atmospheric vibration’. These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European network.”"
- "The risks posed to electrical systems by big variations in atmospheric temperatures are well known in the industry, even if it is rare for problems to manifest on this scale."
- "“Due to the variation of the temperature, the parameters of the conductor change slightly,” said Taco Engelaar, managing director at Neara, a software provider to energy utilities. “It creates an imbalance in the frequency.”"
There's a connected grid across most of continental europe. If Portugal's grid blacked out unexpectedly then Spain's grid could have been made unstable too, which then continue onwards to South of France and Italy.
Just a typical cascade failure because it means everything's now running with lower tolerances.
A cascade failure (of any system) would propagate until it runs into a part of the system that's resilient enough to not fall over.
Several people have said the amount of power transferred between France and Spain is very low compared to the amount of power consumed inside either France or Spain, which seems like as good as reason as any that it didn't affect France. The grid in France was presumably able to absorb the sudden change in power flow without itself breaking.
There aren't as many interconnects between Spain and France proportional to generation as there are between Spain and Portugal.
Also France has a massive Nuclear base load and is usually an electricity exporter so their grid will be a bit more resilient (the spinning mass of massive turbines in Nuclear plants provides inertia to the grid itself) than the Spanish/Portuguese grid that would have had a decent renewable mix.
At the same time, electricity grids will have various measures to prevent instability spreading. Whether that's load shedding (dropping parts of the grid), removing production, or what France likely did, dropping the interconnects. This is usually fully automated.
If there were more connections to France it _maybe_ could have spread further. Nuclear power plants can be twitchy if things go south so if it had spread enough to knock 2-3 french plants offline then that could potentially have toppled France which then as a big exporter could have swept across much of the rest of mainland Europe (and maybe even the UK which might struggle to adapt to the loss from France through its HVDC interconnects if all three were maxed out)
no, most electronics is shielded by default/by law. check UL/CE requirements. but it can make circuit breakers "turn off". EMP meaning nuclear high above, there exist emp devices of size of small family van, but those have range few 100 m/ft
I've seen the assertion made that we can statistically measure how many people died due to (for example) a heat wave, but we can't say for sure which ones.
I'd imagine something similar applies here. You'd have some number of deaths specifically attributable to lose of power, plus countless other deaths caused or prevented in non-obvious ways. This might be visible at a high level as a statistical outlier in the total number of deaths during the time period of an outage.
Well obviously. Computers use electricity, so if the grid goes out so does internet traffic. Local data centers might have backup power but local homes and everything in between the homes and the data center probably won't.
There's the Madrid Open tennis tournament affected. No Swiatek - Shnaider game until they'll get online. The restaurant is lit with candles, but the DJ has power. So there's music. He was clever.
Practical Engineering on youtube has some good videos on the electrical grid. This one in particular is a good explanation on what they had to go through to get power back on: https://www.youtube.com/watch?v=uOSnQM1Zu4w
I hope somebody building and selling microgrids and energy storage systems is ready to take advantage of this situation. Mass deployment of batteries at the grid edge is a great idea for a lot of reasons, but none more immediately compelling than "the power won't go out if the grid goes down."
It should be mandatory for critical infrastructure like ATMs to have 24hr batteries installed. And some sort of power backup to maintain internet access as well - even if it's just a matter of everyone being allocated a small amount of starlink data or public wifi.
These things have relatively small costs, but make the system much more resilient.
My fear is that without access to cash, people might loot stores or riot to get essentials, once they get desperate. Keeping ATMs online for longer would make that kind of event less likely.
Living in Maine makes you prepared for this type of stuff.
It happens every winter. When I was a kid, we once went 2 weeks without electricity.
Wood stove becomes an obvious necessity. Not just for staying warm. For cooking.
Chickens in the backyard too.
Those moments show you that being self-sufficient is an extremely important skill, and while we're not totally self sufficient in those moments, we get pretty darn close.
Really makes you appreciate the pre-electric era.
People cut down forests with axes. Pulled the stumps out with oxen. Cut the lumber with hand saws. Chiseled foundations with pickaxes. Etc, etc.
With the constant threat of Nuclear War, there's a sword of Damocles hanging over our heads that we will return to this era.
That makes it all the more important to learn these skills.
It's fascinating to see simulations of power grid disturbances propagating due to the limited speed of light/causality: https://youtu.be/5vAkdXRyKWM?t=713
This is really interesting. Even though I'm used to reaching websites on the other side of the continent in less than a second, it's incredible seeing how fast a frequency disturbance propagates.
We had power on the canary islands (belong to spain, but located in the atlantic ocean ~80km west of Africa) since it is an independent grid. But couple of hours into the blackout (starting 5-6pm local time), internet issues occured - with confirmed multiple landline providers going offline (Digi, Movistar) while others seems to continue to work (Vodafone). Also some cellular 5G were affected, otheres continued. Interested to see the postmorten - I suspect after 8-10h the data center's diesel generators on the peninsula that route the traffic ran empty. Internet operation resumed during the night (for me around 1am).
I've lost power at 11:30 and regained at 00:12 (Portugal). My hat's off to those brave engineers and technicians that managed to black start the ancient, under-invested grid in record time. Massive respect.
I'm wondering: are there any simulator games in which you have to take care of the stability of a nation-wide power grid? This whole situation has me interested in logistics of power distribution now.
I'm from Spain, electricity is almost fully restored in my area. But my fiber network is still down, same with the buildin elevators.
People do not realise that when the backup supplies (batteries, diesel, whatever...) get drained, a cold start of non electric infrastructure could also be needed because syncornizing a mesh of unstable IT systems is tricky by itself, in some cases needing physical access.
What network are you on? I'm outside Mataro and with Vodafone, got internet connection and power restored some time early morning/late night (after 02:00, before 09:00)
MasMovil / Orange. Yesterday connection went on and off while the router powered by and UPS. But since last night, the fiber does not sync with the GPON anymore.
Ouch, that sucks. Bet their technicians are on overtime already too :/ I'm guess you've already tried the true-and-tested "Turn off, wait five minutes, turn on again"?
Usually we've had lots of issues with our fiber from Vodafone (like random ~70% packet drops from time to time) but happy Vodafone seems to have recovered quickly in this case, no GPON issues here as far as I can tell.
Thats right around when we got phone service again, but no electricity until early this morning. Guess our internet/cellphone coverage comes from BCN which would make sense.
As of 2 min ago..( 14:32 WEST ) in many parts of Portugal electricity still out, and also Water systems also offline. Report is for some it can take up to 10 to 12 hours before power back up. Probably many locals not reporting here as they are low on battery.
For offline messaging over Bluetooth/Wi-Fi without internet or central infrastructure, I saw these recommended by ChatGPT Deep research and they seem current:
Bridgefy (Android & iOS) – direct Bluetooth mesh within ~100 m, instant one-to-one & group chat, <1 min install.
Berty (Android & iOS) – E2E-encrypted over Bluetooth/Wi-Fi; Briar (Android) – E2E-encrypted over Bluetooth/Wi-Fi with Tor fallback.
Manyverse (Android, iOS, Desktop) – offline-first social feed that syncs posts over Bluetooth/Wi-Fi when peers meet.
Can anyone vouch for these or know better?
Whoever downvoted, a comment explaining why would be nice, when the cell tower generators are out, and there is only phone battery left, offline communication options could be useful, I'm glad I'm not in that position right now.
When I Googled it, the apps listed include Firechat, Signal and Briar. Signal is centralised, Firechat is dead, and I don't think Briar are current either? I wish it was a simple web search. Even Deep Research needed the ML pot to be mixed around a few times until it gave an good answer.
Seems like some people are not interested in honest and useful feedback because this got downvoted too! I thought it made clear points about an important topic. Well, whoever sees fit to do this, I hope you get whatever it is you want.
My experience of it in Lisbon was undramatic. I was able to buy medication from the pharmacy. Life continued as usual more or less apart from some inconveniences, people playing cards in the street and a sense of everyone stepping away from the tech. Eyes were rolled as the reflexive suggestions of "putin" were inevitably voiced, more to make conversation than anything else. I met neighbours that I'd never talked to before, and reconnected with some I hadn't talked to for years. People played guitar on their balconies rather than spotify bluetooth speaker pop. The neighbourhood cheered as it came back online but part of me, selfishly, would have liked a few more hours of blackout.
Would have been a very different story if it were a week rather than a day, but I was left with a sense that this complex community of locals and foreigners is stronger than I previously suspected.
Do people forget how to drive if there are no lights? I can't imagine many people would think "well, there are no lights, which means GREEEEEEEN!". Over here, a traffic light that doesn't work means it's now a stop sign.
That's still a recipe for massive gridlock as treating most traffic junctions as stop signs doesn't scale unless there are traffic cops to manually direct traffic
I've never seen a traffic cop but I've driven past maybe fifty or a hundred broken or shutoff traffic lights, mainly in our largest city and never got stuck. Local law states that you give cars coming from the right precedence and the crossing self organises around that.
I work in Barcelona and live in a town 20Km outside it. I drove back home just fine at around 16PM. Some traffic in the city, but manegable. My town has had electricity in homes since 15PM, some neighbourghs told me.
At night I drove to my parent's (different town, 10km away) since I couldn't communicate with them and was worried. Their town was pitch black. People were driving mostly safe and slowly. Of course you get the random POS that doesn't care, but I didn't feel in danger at any moment. My parents were fine. I drove back home later and same story.
I just woke up (it's 7AM here) and the Internet is back and I have a message from my parents, their electricity is back too.
Overall, I didn't see any crazy behaviour.
This complaint needs to be passed to European regulators at either BEREC, ACER or DG Connect, since using a loginwalled site is a violation of EECC directive 2018/1972. Ideally they can just designate 1 approved site that all utilities should harmonize on using for their status updates.
Is this a joke? Because that seems like exactly the wrong thing, and such a system would itself be a bottleneck for downtime issues.
Using twitter has the huge advantage that spikes in users in Spain for checking this stuff is a rounding error in the normal traffic so is very unlikely to take down the status page.
no one would/should prevent companies from using also private channels like X, FB, Instagram, etc but enforcing a public channel that doesn't require private citizens to register, accept T&S and share their private data with 3rd party, unrelated private corporations to be informed of critical, public safety information would be helpful.
Saying things you know to be untrue will not win you political friends in the long run; quite the opposite: it’ll cast doubt on every other opinion you have
That's funny, Fascism has never struggled to gain followers while lying about everything. In fact, an explicit abandonment of truth and reality is inherent in most Fascist regimes, and is also prominent in other types of cult of personality authoritarianism.
Maybe outright lies are more effective at getting followers than you suggest.
That being a distasteful reality doesn't make it incorrect.
There will always be a number of loons willing to support any fringe idea you might mention.
The key is: will the bulk of the people, who are in the center, go for it?
And the answer is: if they perceive the idea as being cynical, or purposely deceptive, then no. Even if politicians temporarily get away with lies, every lie does long-term damage to a movement.
Only for those who don't understand sarcasm or other forms of figure of speech and art in general. For those I'll prefer distance anyway, so it's a win-win.
It's hyperbole, not sarcasm, and it's an effective way to neuter yourself politically. Have you ever rolled your eyes at someone going on and on about communists in the deep state?
You’re kidding right? The overwhelming majority of people do NOT use X or have accounts there. We are in a bubble. (Not making a Musk point here as it was the same before the “exodus”)
I remember years ago someone scanned the Internet IPv4 space for open unpassworded VNC servers. Many of them looked disturbingly like industrial control systems.
They could definitely just post updates on their website, the infrastructure is there, running, if they want to inform people on Twitter it's just a bonus to message it through the network that is Twitter.
But important/urgent updates only via Twitter is definitely a huge no-no.
I assume that a major electric provider has the capacity to build a simple status update system where they post. It's much simpler than any "blog platform" case study used to assess junior engineers. You are assuming that it's their IT systems which crashed the electrical network and there's no sign of that at all at this moment, you are misleading in there to try to score a point, stop.
It's far easier to use Twitter but it doesn't mean it should be used, it fences out people like the OP and me, who do not have Twitter nor want to have Twitter, I don't want to be forced into using a private corporation service to get status updates from the electrical network where I live in. It's quite a simple proposition and very reasonable, not sure why you are so incensed by a quite reasonable expectation.
> Also, why do you assume that website wouldn't crash under the sudden 10000x load? It is an utterly useless solution, that wastes time and solves nothing.
Because it can be cached very easily, it's 2025 where setting up this kind of cache is extremely easy compared to 2005.
> Like does your engineering skills suddenly magically evaporate the moment Elon's name is mentioned?
Please, stop, you are too irrational to understand a very simple and reasonable thing, no need to start throwing Elon into this bullshit, just stop here with the rabid lunacy. I was against major corporations only posting updates on Twitter waaaay before Elon bought it, I still stand by it.
Using the term whistleblower in this manner is inappropriate; actual whistleblowers are individuals who bring to light illicit acts by organizations or governments at great personal risk.
> This is literally the whistleblowers about cashless society have been warning everyone about for well over a decade now.
This is how humans are with all catastrophes–there isn't enough money until after something really, really bad happens and suddenly there is enough money to fix the issue.
NYC is extremely vulnerable to a 9/11 style attack on the fresh water aquaducts. Fuller wrote about this all the way back in the 60s in Operating Manual For Spaceship Earth:
Thus under lethal emergencies vast new magnitudes of wealth come mysteriously into
effective operation. We don’t seem to be able to afford to do peacefully the logical things we
say we ought to be doing to forestall warring-by producing enough to satisfy all the world
needs. Under pressure we always find that we can afford to wage the wars brought about by
the vital struggle of "have-nots" to share or take over the bounty of the "haves." Simply
because it had seemed, theretofore, to cost too much to provide vital support of those "have-
nots." The "haves" are thus forced in self-defense suddenly to articulate and realize productive
wealth capabilities worth many times the amounts of monetary units they had known
themselves to possess and, far more importantly, many times what it would have cost to give
adequate economic support to the particular "have-nots" involved in the warring and, in fact, to
all the world’s ’have-nots."
> This is literally the whistleblowers about cashless society have been warning everyone about for well over a decade now.
Indeed but it stands to reason that this outage will last maybe a few hours until the grid has recovered. A nationwide full blackout is a scenario that's on a "once a quarter century" level, and the last one in 2006 was resolved after two hours. It's Europe, not the US - our grids operate on much, much stricter requirements and audits on resiliency, hell since last year we got an active warzone in the ENTSO-E grid and it hasn't been too much of an issue!
Not much of value will have been lost in the meantime. The only ones who are truly and beyond screwed by such events are large smelters and similar factories where any prolonged downtime leads to solidification of the products which, in extreme cases, require a full reconstruction.
As for "I can't buy eggs in a supermarket now"... lol. People need to learn to chill down a bit. You won't die from having to wait a few hours to be able to buy the eggs.
> Not much of value will have been lost in the meantime. The only ones who are truly and beyond screwed by such events are large smelters and similar factories where any prolonged downtime leads to solidification of the products which, in extreme cases, require a full reconstruction.
I think you've left out a few things, I remember doing on site work at a pharma company that required some downtime on one of their lines and if we went over the allotted time, they would be charging us up to 2 million EUR an hour. Hospitals and critical services SHOULD have backup generators etc, but depending how long this lasts a lot of things can become a major problem.
The majority of the cases will be fine, but when there's mass confusion and interruption like this, there's always horrible stories that come out.
> None of that changes the difficulty of a black start.
That's the beauty of the European grid: it is not a black start event for Spain, at least as long as even a single link to any of the neighbouring countries is available.
The difficulty is still the same: for each piece of the puzzle (substations, high voltage power lines, transformers, generators, loads), they have to wait until enough power is available at the connection point, and carefully manage the power balance so it's neither too much nor too little (and if they get it wrong, things trip offline and they have to start all over), while the grid is in a degraded state (often meaning no alternative paths, so a single fault can set them back several steps). The only difference being that the black start "generators" are these still working international links (which could be very far from the important parts of the national grid).
It might be faster to instead black start several independent power islands in parallel, and connect them together as a final step. At least in my country (Brazil), that's how it's done for large-scale blackouts, even when some of the country still has power; it was done that way for the partial blackout in 2023, and there's a written procedure on how to do it (which is available on the operator website, if you know where to look). In 2023, some areas failed to black start for one reason or another, and had to wait for power from the outside; other areas managed to black start as expected, and were then synchronized with other areas until everything came back together.
The grid is not just an aggregation of individual sources and sinks; it takes active effort to keep them all working together in a useful way without just collapsing again into another cascading failure. For that reason, your solar inverter doesn't come on until the grid operator wants the solar inverters to come on in your section of the grid.
It's tempting to think of the grid as something grid operators control, feeding power from point A to point B, but the grid is actually largely uncontrolled - the power just flows wherever it wants to - and the only controls they have are turning on and off generators, adjusting their throttle, disconnecting loads (rolling blackouts) and sometimes opening circuit breakers (though this is not normally useful). They don't even have precise real-time monitoring of the whole grid - only specific measurements in specific locations, from which the rest is estimated using lots of maths (which is how you would design it too, if measurement devices cost $100,000 apiece). That's why it's not a trivial task to keep it working.
However, you're able to have your own, private miniature grid, on which you can power your own loads from your own generators. It's even possible to do this with solar inverters! You will need to specifically seek out this capability, and get extra hardware installed, which is probably why you don't have it. You need a "transfer switch" to definitively disconnect your private grid from the main grid when you're using your private grid capability - it's not allowed (and not safe, and will blow up your equipment anyway if you force it) to just feed power onto your local unpowered section of the grid.
Solar inverters could simulate inertia by not running at 100% most of the time, but you do want the free energy machines to run at 100% as much as they physically can, because it's free energy, which means there's no buffer for simulating inertia. It's been commented many times that batteries can be used to simulate inertia. You can also literally just add inertia, in heavy spinning lumps of metal that don't do anything.
Germany has dozens of links to its neighbours. We don't need much in terms of "black start" capacity, that's just pointless fearmongering by fossil fuel and/or prepper propaganda sites.
As long as even a single link to any of our neighbours is up and running, it can be used to start the rest of the grid - which is exactly what was done in the 2006 outage and why that one took barely two hours to be resolved. The only truly screwed country at the moment is Portugal because all their grid links run through Spain.
> As long as we’re throwing shade at EU vs US, I can’t remember the last time the US had a nationwide blackout, certainly not in my lifetime!
Talking about "national" in the sense Spain (pop. 48M, 506,030 km²) is roughly equivalent to a few US states. A similarly (population/area) sized outage occurred a couple of decades ago:
Indeed, and that’s the main problem: you can’t remember or know anything.
It’s is a known fact that in general the US power grid is orders of magnitude less reliable than in Europe. And the excuse of “the weather is more extreme” is just that: a lame excuse.
Just count the number of American households that have generators and/or batteries vs the Europeans if you really have an honest desire to know anything about anything.
The US has three “independent” grids so losing them all would be hard. But I believe at times Texas has gotten close, and East went pretty dark at some point recently.
CA of course has rolling blackouts for other reasons.
2021 would have been a non-event if people in Texas weren't propagandized about some nonsensical "Independence" bullshit.
A few more interconnects with the rest of the country and it wouldn't have even made the news.
this is after decades of Texans bragging about their independent power supply. Many Texans still believe outright lies about the blackout, like it being "caused" by green energy sources, which was false.
It was caused by free market participants not spending capital to harden their network. Solar panels and Wind Turbines work great in the cold climate of Canada.
The storm that caused such a problem is a once every ten years storm. The grid companies all should have foreseen this with even minimal investment in planning. They didn't, because that's less profitable, and the "regulator" in Texas has no ability to punish them for pinching pennies on reliability and resilience.
There's five different grids in North America (Eastern, Western, Texas, Alaska, Quebec) so something would have to go very wrong for a nationwide blackout.
As was pointed out, the USA has three independent grids (east, west, and Texas) and EU countries are roughly comparable to states (except with less federal power). The equivalent of a European nationwide blackout would be a US statewide blackout, and those HAVE happened, definitely within your lifetime if you're old enough to use Hacker News, mostly in Texas.
I was on a plane during the Northeast Blackout of 2003. Landed, got in my car, and attempted to leave the parking garage but with no power the automated EZ-Pass payment wasn't working nor were the credit card machines. Most people, myself included, had neither sufficient cash nor a checkbook on hand. Huge logjam of cars. The workers ended up getting the old school handheld metal credit card machines that created an imprint of the credit card on carbon paper from some long forgotten storeroom and using them.
Many years ago I worked in a Safeway grocery store. We would have occasional power failures that would leave the entire store dark; we would all be given flashlights to help customers find their way out.
The cash registers, though, had backup power, so the store could still take their money.
I worked as a cashier at a large UK supermarket when I was a student. In our training we were told that if there was an outage with the cash registers we should ask shoppers to estimate how much their groceries cost and accept what they told us. Payment could be by cash or cheque.
Apparently when this had been done in the past shoppers were generally honest & relatively accurate.
Many situations call for pen-and-paper backups. Giving out receipts the old way should, in theory, be a possibility, then backfill the computer system later.
Sure, but if we are talking about backup in outages you can also get credit card payments when power is down using something like a Stripe Terminal.
This is actually exactly the case that I had in one trip to Andorra: the power was down for 2 hours while we were choosing equipament for skiing. The shop had no issues getting our orders done though, because they just manually filled the orders with pen-and-paper and did the payment with a credit card terminal connected to a smartphone.
In my experience, for the 4G/5G network to be down something really serious must be happening. I had long power outages (more than a few hours, in some cases even days) that affected multiple regions in places that I lived before that still had working cell network. I assume cell networks have backup power and preferential usage of the power grid, but I am not a specialist.
And I am not saying that you shouldn't accept money as backup, of course you should. But what I am saying is that you can still accept credit cards even during most power outages.
Same as Software Engineer, it is impossible to have perfect, 100% reliability, but it doesn't mean we can't improve from 99% to 99.9%, for example, to have a better service.
Used to work for a telco. Cell sites have battery backup. Some have generators. Any fibre repeaters also will, as do any radio based backhaul sites. The HLR/core network etc will run indefinitely due to generators & strict fuel supply contracts for said generators.
If a city has an extended power outage such that the battery backed cell network goes down, then everything else will be failing too and payments are the least of your problems.
Without electricity the water system depressurized, which contaminates it. After about a week the sewage pumping stations have backed up so the sewer system is starting to fail.
Modern cities cannot operate without electrical power given their scale and density.
It is bizarre to think the biggest problem is "how do we keep a transaction of value?"
Like, just declare an emergency and let business owners be reimbursed by the government.
Credit card processing existed before widespread telecommunications infrastructure. Maybe we should require payment cards to have raised numbers like they used to so the old carbon copy machines continue to work.
Credit cards and payment networks have always explicitly supported "Offline" processing like that.
The kind of fraud that system enables isn't really common.
In Lisbon's airport they are temporarily back to just stamping the passports without the biometric stuff. There are already reports that the checks are being lax.
I was in a Tesco when there was a power outage - the self checkouts all died (and rebooted into a custon Debian based OS with a Chromium front end by the looks of it), but the staffed tills still worked and could accept Chip+PIN but not contactless.
> My supermarket, for example, only has electronic cash registers. And no price tags.
I know someone who works at a supermarket, and (some of?) their point of sale (POS) systems have a small UPS that can run for a couple of hours to ride through smaller outages.
I used to install those systems, and it's a couple minutes at best, if they've replaced the batteries recently, which they never have. It'll give them time to finish the transaction they're in the middle of ringing, then shut down cleanly, because the server in the back room _should_ have a bit more battery to keep the database consistent.
PoS systems aren't particularly power-hungry, but store owners never want to spend an extra cent, so they go with the smallest UPS they can manage. (And arguably if they went with a big overkill UPS, its after-outage recharging power would be larger so you'd be able to put fewer registers on a single circuit, so it's not as simple as just dropping in a bigger UPS.)
Very good point that nowadays many stores rely on barcodes and product IDs to get prices, and don't label individual items... So even pen and paper to keep track of takings is no use since they can't even figure out prices if the system is down!
> My supermarket, for example, only has electronic cash registers.
That's insane to me, in the EU anyway it's not permitted to only accept electronic payments..
> Retailers cannot refuse cash payments unless both parties have agreed to use a different means of payment. Displaying a label or posters indicating that the retailer refuses payments in cash, or payments made in certain banknote denominations, is not enough.
They have to accept cash in the US as well. The post you're replying to is just saying that they can't ring anything up or accept any payment without power.
I just look to Gaza strip as an example how society continues operating under extreme adversity from the western powers, incl. total lack of electricity distribution for more than a year and a half and full blockade of water, food, and medicine.
I don't think this is true, and I would like to see an explanation of how it might be that quantifies the effect: are we talking electric field strength or magnetic field strength? Modern parts are generally better at ESD than old ones, as well.
gas stations, at least in europe, have "safety zone" around them.
also depends on data center, some datacenters are directly in middle of european city. in most dense part of city and dense in europe or asia is something else then dense in america.
I agree that bitcoin is a poor solution for anyone but a small elite. But as long as you have the mobile network on diesel generators (which you do in countries where power cuts happen regularly), something like M-PESA still works.
Without power or an internet connection you can't validate that the btc hash is unspent, avoid double spends, or validate that your taking ownership of it made it to the blockchain and was confirmed for a few blocks.
Sure you can transfer the private key from one device to another, but (a) you can't know the other person didn't retain a copy of it and (b) you would be limited to spending the exact amount you have in an existing transaction because you couldn't send a transaction to the chain that splits it.
You've to got a reputation system with identities also stored on the blockchain, and you accept the transactions only from the most trustworthy people. Counterfeit paper currency exists, and counterfeit bitcoin (not BTC) transactions might exist, but not every person is equally suspect of using counterfeit money.
Sure, but then you aren't really using bitcoin. The whole point of bitcoin was to be a trust less system where transactions are verified cryptographically on the blockchain.
Transacting bitcoin private keys without going to the network and trusting the other party to not scam you defeats the whole purpose.
You can use cash but it mostly does not work in case of big chains or stores. They need to have access to their SAP/ERP software...It is not about payment for a long time.
Running a generator to power a point of sale machine and maybe an offsite server is a lot easier than generators to power the whole electronic payment chain.
Let me introduce you to the physical receipt block which is made of paper and can be filled up with a pen, and what is still often used in, for example, Christmas markets.
Depending on the country it’s not that easy. Maybe something changed in the meantime but where I am those blocks are prefilled-numbered and stock needs to be controlled.
It’s not like you can (could?) keep a block „just in case” and thus many shopkeepers wouldn’t even bother in case of outages.
Depending where you live a good old trust can be a currency. Humans are great when it comes to adaptation, I bet I could just write on paper name, CC number and leave it on a paper for shopkeeper and everything would resolve just fine..
I hear you, and in my second job 20+ years ago we were still trained (in a cursory fashion) to use physical charge slips in case of a POS system network outage, a full power outage would strain my imagination for ability to transact business at the majority of stores other than small mom and pop operations, and tbh even if such an outage had happened to me in 2005, I would not expect my department store to remain open and conduct sales.
Most retail workers are GenZ and struggle to understand what this would even look like because they’ve never conducted any transaction without POS computers (for looking up prices, for tallying them, for figuring tax and total, and computing change), so even if a dusty manual in the stockroom technically spells out a method of ringing sales using nothing but pen and paper and maybe a solar calculator, I would be surprised if any of the clerks working any given day would have the initiative to initiate an offline protocol. Most likely the store manager would usher customers out, lock up the store, keep the staff for 30 minutes to see if it came back on, and then go home.
> How do you open the cash register for change when power is out?
I've only seen a few but I believe they have springs on the inside and roll on little wheels similar to how desk draws roll. Most can be opened with a key to trigger that event.
Many countries require receipts to be sent to the tax services instantly. In many cases a long enough power cut (days) would render all transactions illegal
(And in many cases you cannot legally pay large amounts of money in cash, it has to be electronic)
Which countries exactly? I've traveled in UK and France and in both countries when the online cash register was down, they opened their physical ledger, made an entry and gave me a physical handwritten paper receipt written with pen and ink. They said they would make enter the same data online when the online cash register comes back up.
My understanding is that this (NF525) only applies to computerised cash registers (software must be certified NF525 compliant), which there is no obligation to use in the first place.
So it is perfectly legal to use pen and paper and a cash box.
Most governments are not as stupid as the anti-government wingnuts would have you believe. They will not prosecute you for taking transactions non-electronically during a power outage.
In some countries whenever you print a receipt, a copy is also sent to the IRS equivalent of that country. Obviously there are events where that can't happen due to technical reasons outside of the store's control.
Which countries? And, again, I doubt that this is the full picture because there are many cases where people simply don't "print a receipt" perfectly legally...
Germany for example mandates printing a receipt. The receipt must be stored in a certified storage inside the cash register and is signed cryptographically, including the hash of the previous receipt such that there is a hash-chain of printed receipts. Therefore each printed receipt that the customer takes home (and maybe at some point hands in to the tax office for some reason) can be used to check the integrity of the cash register storage and all prior receipts in the chain.
Same in Portugal. Sync with the tax authority can be immediate or deferred (every x days). Obviously, you can still invoice manually using a receipt book, in case of failure or unavailability of software systems.
Thanks! European red tape madness strikes again... At least in France cash registers are not mandatory (for now...) so there is a way around this madness.
This is a bit more than just red tape madness, it's a strategy to make businesses more transparent.
This is about trying to reduce non-reported transactions and too many people dodging their reporting. Even if the rules for cash registries and reporting are detailed,
a) that's not really expensive for businesses - it's easy to automate and there are quite a number of competitors;
b) compared to accounting and tax rules, they are dead simple.
Receipts or invoices are the basis for a firm's whole economic activity, including the underpinning of their financial reporting, their tax burdens etc. And businesses failing to provide receipts erodes not only the tax base, but also any rights a consumer may have.
Its actually less red tape. Getting a second copy of a given invoice is trivial, processing of invoices for tax rebates is also mostly automatic (such as health and education expenses); tax invoicing uses well-defined formats, so its trivial to migrate between systems, and perform all kinds of analysis. Also, it increases transparency - you know that eg. the VAT you're paying is not ending in the vendor's pockets.
No power is different than no internet, so that was different. We were a 24/7 store (7-Eleven) and sometimes we would close if it was a prolonged power outage. No one has 100% uptime, not even 24 hour gas stations.
Why would an employee tracking device be necessary to make transactions? It is just people without a cash position performing mental gymnastics in an attempt to justify it, by trying to bring it down to the level of electronic payment methods they convince themselves that, in a similar event, they would be no better off with cash, when in fact they almost certainly would. A merchant who refuses cash on account of a disabled cash register, under these circumstances, is not a serious one, since to do so would be to refuse all business. Those merchants who are so encumbered by scale and bureaucracy that they would be completely incapacitated should be circumvented.
> When the power is out one cannot pay with cash either - because the cash register is offline.
Cash registers can be connected to small UPSes to ride through smaller outages. You wouldn't need a larger battery if all you want to do is ride through a few-hour outage, or even a whole business day (8-12 hours?).
I used to live in an area that regularly had power outages during thunderstorms. I was also once in London during a power cut that took out most of the city centre. It was a strange experience, but not as weird as walking through empty streets in the time of covid.
Maybe just my perception, but power outages seem to be getting rarer with time, though when they do happen they seem to be far larger.
We had one in my country when I was a kid 30 years ago after a substation tripped and took down the rest of the grid. We broke out the candles and chilled I guess. Power was back up in my area that same night.
Ok...? Is there like more to this narrative or are we supposed to buy that russia just hates all of europe for being europe now? Whatever happened to the "rational actor" canard?
Russian establishment DOES hate western civilization, that is not "narrative" that is fact told by them in live tv to your face FOR LAST 70 YEARS. That is not "narrative".
Western style of life is not only EU but also USA. I do not know how people can even doubt this lol.
If they wanted western life in russia, then establishment will make changes to have it there, no ? Russia is NOT democracy, it is tyranny, autocracy. Again it is not narrative it is what they do there lol
I don't buy this argument, it's the pinnacle of selection bias.
`any != all` after all.
Your argument is essentially; because some Russian people send some of their children to be educated or buy some property in the west (as a portfolio of how many?) that the argument that the state of Russia dislikes the EU holds no water.
To me, it's hardly evidence of anything, just like how some people in the UK fetishise Russia- yet the UK government is actively hostile and condemns without hesitation- Russias actions towards Ukraine.
My argument is not just about "some people", people who are pretty high up in the hierarchy. How about russian ex-president?
The "hate west" narrative is pushed because it makes sense during the war. If Putin decides now praising the west will let him keep the power the propaganda machine will do a 180 turn
so they HATE west no matter what they say, so you are correct in that.
but you are making wrong conclusion,
machine is not bad thing BUT they are good people.
They ARE bad actors no matter if they use propaganda machine that way or any other way or not use at all. they are bad actors period. propaganda machine is separate thing.
They hate west because by western standards they will be in jail for all LOOTING of russias natural resources and killing and abusing their own citizens. West is punishing people who do bad things. Russian oligarchy kids did nothing wrong, presumably.
contrary to west, in russia you get beaten by police because your children in west posted something on Xtwitter...
Alexei Navalny, Boris Nemtsov, Boris Berezovsky, Sergei Magnitsky, Stanislav Markelov, Anastasia Baburova, Natalia Estemirova, Anna Politkovskaya, yuri Shchekochikhin, Vasily Melnikov , Vladislav Avayev , Sergey Protosenya, Yevgeniy Palant, Yuri Voronov, Ravil Maganov, Vladimir Sungorkin, Anatoly Gerashchenko, Vadim Boyko, Vladimir Makei, Grigory Kochenov, Vladimir Bidenov + Pavel Antov, and thousands of others.
most spectacular was - Pyotr Kucherenko where two men holded him and third put shopping bag on his head, and noone saw nothing in whole plane... except three photos were taking of incident...
Europe is helping Ukraine it its defense against Russia.
Russia has sabotaged a lot of things in multiple EU countries, including Spain.
It's not far fetched to imagine Russia being the root cause of this, or being implicated in some way. Even if they are not, they 100% are watching this closely and learn how they can disrupt power throughout Europe.
I'm not saying Russia did it, just that they could be the cause of this.
Weakening Europe is in their interest, and they already have blatantly done smaller scale sabotage. It wouldn't weaken their position, as you said, Europe is not really interested in taking Russia seriously atm. Sabotages, killings, politician corruption and public disinformation have been common tactics for them for a decade now.
Now that Russia's #1 enemy is under their control, I'm not sure they are afraid to take on Europe more directly.
The EU is the main provider of financial support to Ukraine, the country which Russia is currently attempting to invade, along with one of the major provider of weapons and training.
Well, it's been for quite a while now. Acts of micro- (or mini-) aggression like fires, explosives in commercial shipping services, broken sea cables just became daily news.
The only positive aspect of this is after the root cause is found, the grid will become more resilient in the long term (but these kinds of changes typically take long time).
Again, how would these actions do anything but weaken Russia's position given the EU's apparent willingness to stay on the sidelines? Wouldn't ukraine benefit the most from the perception that Russia is at war with the EU?
For the record, I don't think that this particular event is a Russian act of sabotage.
But Russia is an aggressive authoritarian state that was already caught for (smaller) acts of sabotage in EU, some of them quite dangerous. Why they are doing this? Who knows, war in Ukraine was not rational too. Perhaps some people want to be evil just for the sake of being evil.
As a Russian emigrant, I long stopped trying to rationalize Kremlin decisions. Why authoritarians are authoritarians? Who knows. Mad with power or something.
The strategy is not even a secret. Russia sees itself as a major influence of the Europe.
You cannot control stable governments, so you destabilise them with various tools for prolonged periods of time and then you end up with a country which is much easier to influence.
Outside of discussing whether Russia is behind this or not, the broader Russian strategy seems aimed at undermining trust in European governments. [1]
The goal would be to create enough pressure from people - frustrated by problems like power cuts — so that governments must withdraw their support for Ukraine.
Any "WW III" fearmongering is similar : intimidate everyone into withdrawing support.
Many European countries have created emergency guides to help citizens preparing for crisis like this one. [2]
This, I guess, has the underlying goal of maintaining trust in European governments.
> given the EU's apparent willingness to stay on the sidelines?
... Wait, how are you defining that? Much of the EU is about as close as it is possible to be to being at war with Russia without actually sending in troops.
Good place as any to ask: do you need a VPN to access the “Western” internet? Is the block on the Russian side, or are Western websites blocking Russian IPs?
No in most cases. Meta products are banned, twitter, discord and youtube (this one mostly works in reality), but pretty much everything else is unaffected.
Some Western side companies banned Russia by IP's like Intel, but in general, my list of websites to tunnel through a VPN is rather short, like a dozen and mostly to unblock youtube as meta and twitter are cancer anyway.
The block is on the Russian side (in most cases), but not all Western sites are blocked (Hacker News is working). Most of Russians know how to use VPNs, though it is extremely inconvenient.
Why is that? We're not saying it's definitely Russia, but exploring the possibility they could be behind this.
After the multiple sabotages, killings, corruption, as well as the invasion of a neighbor country, we have some reasons to think Russia is a bad state actor.
Some more-unhinged ultranationalist elements of Russian societies loathe Western culture and what they see as 'decadent' Western values on their culture. This is not new. Xenophobia and hate of the Other has a long and sordid history in Russia.
Moreover, they are unable to just live-and-let-live and actively go out of their way to make other peoples lives miserable. This is due to pervasive zero-sum thinking in Russian strategic thinking. They are fixated on the idea that in order for Russia to 'win', others must suffer and lose.
Spain underinvests in its military. Russian mafias also control much of the costas.
I would surmise that the Russians think that Spain and Portugal are cowed, and want to keep them intimidated and prevent them from increasing their aid to Ukraine.
Some people close to power in Russia (Dugin) actually seem to believe Russia natural range is from Lisbon to Vladivostok.
"Putin channels ultranationalist discourse, such as the Izborsk Club and the neo-fascist Alexander Dugin, in calling for quasi-religious rebirth of Russian dominance, an agenda that seeks to swallow “Little Russia” into a renewed Russian empire that stretches from “Lisbon to Vladivostok,” a phrase popularized by Dugin and repeated by Putin."
I often have the same thoughts as you because, based on what I’ve seen, their actions aren’t strictly rational. For example, the damage to undersea cables will just be repaired, and it only end up angering everyone. Sometimes they also start local fires. I don’t really understand it either.
I do not really think that this needed to be a russians work tho. Spain and Portugal are really kinda far and it would be massively idiotic move even for them.
Noth Koera doesn't have any beef with Europe. Whereas the Russian propaganda says quite shocking things about Europe every now and again, not to mention their previous prime minister that is suggesting nuking European cities on Twitter.
Genuine question: how is it that Ukraine’s electricity grid still works to any degree despite being bombed for years and no doubt intentionally sabotaged / cyber attacked.
But a whole country’s grid can go down like this in an instant?
The answer is simple: EU and USA taxpayers billions pumped into Ukraine to buy hundreds of thousands of mobile diesel powered generators and fuel for them, and spare parts and replacement blocks for everything that’s been destroyed.
On top of that, Ukraine inherited a lot of nuclear power plants, and despite losing the largest one in Zaporozhie region, still operates all the other as Russia doesn’t attack them in any serious way.
Except for the mountain of evil, violent, underhanded and illegal stuff Russia keeps getting caught doing, and has so far escaped scot-free due to Western cowardice?
No, they were suggesting attacking it based on speculation about this specific event, so yes out of thin air.
I agree that Russia is evil, but saying that it got away with it Scot free is just insane. In fact it even plays into Russian propaganda, to say that nothing really harmed them even after the past 3 years of international sanctions and consequences.
And western cowardice? Apart from nuclear war (which I'm glad people are being "cowardly" about), what do you suggest? It's also funny when war mongers online talk about cowardice. Why dont you go volunteer in Ukraine if you're such a non coward?
> It's also funny when war mongers online talk about cowardice
> Why dont you go volunteer in Ukraine if you're such a non coward?
This is an impressive feat. You've managed to package not one -- not two -- but THREE Russian propaganda talking points into a single paragraph. I believe I've just witnessed the second coming of Alexander Pushkin.
And you've managed to pull the typical online war mongers deflection of calling me a Russian shill. Let's say you're right, I'm a complete Russian shill (nevermind that I completely abhor the Russian regime since their intervention in Syria, and think that Putin belongs to the Hague tribunal).
Are you going to actually dismiss the fact that yes, nuclear war would be a very likely possibility if we declared war against Russia? Much more likely than... Not doing that?
Or are you going to use the Reddit argument that somehow, it just won't happen because trust me bro, and only Russian shill would care about getting nuclear bombed?
And yes, it's cowardly to call people cowards because they won't push for a war that you yourself won't fight. I didn't bring up cowardice first by the way, but I guess it's fine to do it when you agree with the person lol.
I'm not. But you are the one going around accusing people of being Russian shills, I hope your little red scare gets better and you move on from Redditor style arguments :)
The thread I answered in had someone playing dumb as if there would be no reason whatsoever to consider Russia a hostile actor, I was responding to that.
That's not what you said though. And it's still a baseless accusation. It's similar to accusing Israel or the US of every single coup d'état or sabotage action just because they have a huge history of doing that. In both cases it's completely baseless unless there's evidence.
That's exactly what I said - Russia is a hostile actor that has given plenty of reasons for war, regardless if they're involved in this, which neither me nor anyone else knows.
War by who? Who do you think should declare war on Russia? And as another comment said, are you volunteering? You can even wage war against Russia now if you want to, volunteers in Ukraine are always welcome.
By the international community of course. The same one that has heavily sanctioned Russia.
> And as another comment said, are you volunteering? You can even wage war against Russia now if you want to, volunteers in Ukraine are always welcome.
That's just a cheap deflection. Me dying in the trenches won't change anything; a coordinated properly aggressive response might teach the war criminals at the Kremlin that they will not be let off. It's just a repeat of Munich/Ethiopia, and no individual at our level can do anything about it but advocate for change in policy to our governments.
It's not a cheap deflection. You want other people to die in the trenches or worse, to die in a nuclear war. The deflection is to say that "your life won't change anything", which I mean sure yes, but that's what every single war mongers that doesn't want to fight the wars that they promote say. If everyone said that, no one would be fighting against Russia right now.
And it's funny to say that it's a repeat of Munich / Ethiopia. Again, war mongers will always prefer comparing to the few times where war was the only solution. The only issue is that not only do both situations have nothing in common (not that anyone cared about Ethiopia, since European colonialism in Africa was nothing new), but it also erases the fact that we are talking about nuclear powers here. If Hitler had nuclear weapons pointed at London and New York back in 1938, then obviously the equation wouldn't be as simple as "stop him now" even with hindsight lol.
> You want other people to die in the trenches or worse, to die in a nuclear war
Other people are dying, and more people will until the aggressor is stop. The aggressor has shown loud and clear he has no intention of stopping.
> Again, war mongers will always prefer comparing to the few times where war was the only solution
Because the situation is comparable. Putin has clear and overt ambitions for further expansion (one of his minions already published the plans with Moldova next). If he isn't stopped, like he wasn't in Georgia and Crimea, he'll just continue. So the comparison is very apt.
Yes, nuclear weapons add a serious nee dimension to the problem. That doesn't mean we should let Putin get away with war crimes.
It is quite common to see bad geopolitical actors paying people to go online and accuse Westerners of being "Warmongers" or telling people who object to Putin's wars of imperialist aggression to "go and enlist and fight" with the subtext that the audience are weak and decadent Westerners with no stomach to fight a literal Mafiya-run fascist dictatorship.
Speaking strictly for myself, given that I want to avoid having people think I'm apologising for the Putin regime, I do not ever use these arguments in good faith.
Nope, that's not what I said. I'm a westerner, and it's not about "good faith". It has nothing to do with behind decadent or lazy.
It's because the modus operandi of war mongers, since before world war 1 has always been to agitate for wars that they don't want to fight for themselves. We are talking about nuclear war here, the bad faith argument is to imply that being against it is a sign of being a paid Russian shill. I guess the Pentagon is full of said shills, since they have never even hinted towards a direct war against Russia, or wanted to get directly (by means of troops on the ground) involved in one.
Now I agree that sometimes it's simply not realistic to go all "well do it yourself then". But in this case, Ukraine accepts, trains and equips volunteers. It's not a hypothetical.
Europe still has many megawatts of solar with inverters that disconnect when detecting any grid disruption.
This is the absolute worst thing to do when there is a shortage of power - you immediately make the shortage worse and more grid disconnects.
The real fix is a grid with second by second pricing based on system frequency, and every individual user allowed to set a daily 'spend cap' of euros/dollars, letting them choose how much they are willing to pay for reliability.
Such an market has a huge stabilizing effect on demand, meaning a major incident would probably only have fairly small impacts on system frequency and embedded solar wouldn't disconnect.
You require inertia in the grid to maintain frequency and other stability stuff.
Solar PV is great but is mostly grid-following so cannot operate on it's own. As I understand it you need a minimum fraction of power generation to be large spinning turbines.
I think this problem can be mitigated with add-on rotational mass style kinetic energy batteries or something like that. I don't think variable energy pricing will help if it's an issue with over-demand the grid managers can do rolling blackouts to manage while fixing the supply problems. The grid is just broken at the moment and the solar can't maintain the grid alone.
Large Inverter Based Resources (IBR) such as huge solar parcs, grid-scale batteries or high-voltage direct-current (HVDC) lines can be programmed to behave like rotating generators, or even to smooth out smaller ripples. They also don't necessarily need a leading grid frequency but can be used to generate their own frequency normal to cold-start or resync a grid.
Only "small stuff" IBRs need a leading frequency from the grid and disconnect outside their safety corridor because those usually aren't controllable from some central grid authority. Thus the stupid-but-safe behaviour mandated for them.
That is basically what happens in South Africa today. There are rolling blackouts and rich people with backup generators and batteries are unaffected. It's gotten to the point where richer cities (like Capetown) have their own power sources to mitigate the problem.
The current system is that everyone pays equally for X amount of stability, and for some people they don't get as much stability as they desire (eg. These people in Spain), whilst others pay more money and than they'd like when they would be fine with 5 minutes outages once a year.
That made sense before technology became available for everyone to make their own choice - but that is no longer the case.
Let's skip the technical problems in your theory and focus on the social.
People need power to survive. You know, food, hot water, light, work, internet, mobile phones, entertainment, etc. This requires stability, not second by second pricing.
When you put a chicken in an oven, you want to cook that chicken and eat it, feed your family. Electricity price rising in the next few minutes would mean that you either have to risk disease (chicken staying in the dangerous temperatures until the electricity price drops) or being hungry and throwing food away. This is not how you want society to function.
Believe it or not, but maintaining an electricity grid is a massive undertaking, and the people in charge of it knows the topic much better than you do.
The problem isn't a market problem, it's a physics problem: having a synchronized grid of AC current with many producers over a wide area is a real challenge, even when the underlying issue is resolved it takes a lot of time to add the power plants (or renewable equivalent) to the grid because they must be synchronized.
The fact that the post you reply to includes such technical details as frequency-based pricing, indicates that the author has an above-average understanding of the technicalities of the power-grid.
Also, nobody in the field disagrees that in the more distributed grid we are seeing today, more endpoint communication and control could lead to more resilience. Whether pricing signals are the best path is a more open question, but they certainly appear to be a feasible option.
> The fact that the post you reply to includes such technical details as frequency-based pricing, indicates that the author has an above-average understanding of the technicalities of the power-grid.
No it doesn't. The fact that it's being said in a comment full of nonsense tells me that they don't have “above-average understanding”. They probably have read something, once, and now thinks they are an expert, that's literally what Dunning-Kruger is about.
They seem to believe that the equilibrium of supply and demand is all that matters, when it's just one piece of the puzzle and among the easiest to manage. Large, nation-scale, failures like this one are very unlikely to be caused by a lack of supply alone and markets are nowhere near fast enough to help preventing these.
The reason why we interconnect grids has nothing to do with politics and everything to do with physics, politics can sometimes lead to disconnections (like how the Baltic states disconnected from the Russian grid earlier this year) but it comes with great cost and involves careful planning (the fact that the Baltic states remained connected for almost three years after the start of the Russian invasion of Ukraine should give you an hint of how challenging it was).
The bigger the grid, the more efficient and resilient it is (and managing electric grids on islands is a nightmare), but it comes with a significant complexity and means restarting from zero is harder.
Interdependent grids are usually good: they allow you to average out the effect of a single power station failure over a much larger area, and to amortize prices from a wider area of suppliers.
Don't you realize that the smaller the grid, the more important the instantaneous load variations can be in relative term and the harder it is to keep things running smoothly? It's not a theoretical concern, it's why electric networks on islands are much harder to work with and much more prone to collapse than bigger networks.
It is not. The bigger the network, the more stable it is, that's why countries keep interconnecting with each other despite political tensions between them (no pun intended).
EU electricity market is very similar to it, but interconnected and probably more free market style. A lot of people pay the spot price depending on county and their provider of course.
I didn’t really find anything problematic from an economic standpoint with the Texas power disaster. On the surface, the main problem there was just that a bunch of people were on spot energy plans, in order to save tremendous amount of money as long as the spot price stayed low. This is simply gambling. Those people after some period of savings suddenly cried foul when their gamble went south as prices spiked. The only issue I see here is if they weren’t adequately informed of that possibility. Obviously, the people who got the $2000 electricity bills are going to be the decide they weren’t informed adequately, and I’m sure it was spelled out in a document the size of an iTunes license agreement that literally no customer ever read. But anyway those people had themselves to blame as that plan was not mandatory or default.
This is the exact kind of techbro nonsense that I’ve come to expect from this website. No empathy for your fellow humans and dressed with a side of “computers and code plus finance”.
I thank the heavens that the people who run the electricity system do not share your opinions.
Thanks for the clarification. However the idea behind this hypothesis is that the inertia stored in physical generators is mostly responsible for keeping the grid stable and as less inertia exists in the grid the risk of this goes up.
Good enough for what? I've read the article. It's speculating that solar could be the culprit as it was the main source of energy at that time, but the grid operator couldn’t say what exactly failed, only that they had "two separate losses of power generation in southwestern Spain, a second and a half a part".
If you look at the generation stats[0][1], everything from nuclear to gas to solar dropped. And it was solar and wind that kept producing most of the energy for hours after the failure.
If renewables were the cause of this blackout, then fine, but that's not what the WSJ article says. The guy who wrote the tweet you linked to was blaming renewables 2 or 3 hours after it had happened and the tweet contained wrong information, as pointed out by a different comment here. To me, it's not a good source.
Just to be clear, if renewables were the main reason behind this blackout, then let's blame them and point out their weaknesses and see what we can do to mitigate them (batteries?). What we shouldn't do is jump to conclusions before we know what the heck happened.
I don’t think the argument (at least the one I’m curious about) is about renewables causing it but that with a growing portion of the grid coming from renewables a black start becomes more and more difficult.
I understand the physics behind this claim although I’d add a huge caveat that I’m not a power grid expert so put the comment here to see comments from experts
> Spain's electricity grid operator Red Electrica has confirmed power outages across the country.
That’s why you still need a strong diesel/diesel-electric locomotive fleet, imagine if Spain had been right in the middle military mobilization and military materiel transport, an event like this one would have stopped then dead in the tracks had they been relying only on electric locomotives.
No, the armed forces have their own fuel locomotives. I imagine any sizeable military does the same thing. Plus most (all?) cargo trains still use fuel. 100% electric is just passenger traffic.
Maybe I'm looking at the wrong article, but there's this:
> For journeys outside the base, *the Army uses Renfe locomotives*.
which, in my understanding, means that in order to move military materiel (to the borders with France, let's say, or to the closest sea-ports most probably) and tens to hundreds of thousands of mobilised men the Spanish Army does indeed rely on Renfe locomotives, i.e not on their own.
This sounds big enough to require a black start. Unfortunately, those are slow and difficult.
If an entire nation trips offline then every generator station disconnects itself from the grid and the grid itself snaps apart into islands. To bring it back you have to disconnect consumer loads and then re-energize a small set of plants that have dedicated black start capability. Thermal plants require energy to start up and renewables require external sources of inertia for frequency stabilization, so this usually requires turning on a small diesel generator that creates enough power to bootstrap a bigger generator and so on up until there's enough electricity to start the plant itself. With that back online the power from it can be used to re-energize other plants that lack black start capability in a chain until you have a series of isolated islands. Those islands then have to be synchronized and reconnected, whilst simultaneously bringing load online in large blocks.
The whole thing is planned for, but you can't really rehearse for it. During a black start the grid is highly unstable. If something goes wrong then it can trip out again during the restart, sending you back to the beginning. It's especially likely if the original blackout caused undetected equipment damage, or if it was caused by such damage.
In the UK contingency planning assumes a black start could take up to 72 hours, although if things go well it would be faster. It's one reason it's a good idea to always have some cash at home.
Edit: There's a press release about a 2016 black start drill in Spain/Portugal here: https://www.ree.es/en/press-office/press-release/2016/11/spa...
In another life I worked as an engineer commissioning oil rigs and I’ve seen how tricky even a small-scale black start can be. On a rig, we simulate total power loss and have to hand-crank a tiny air compressor just to start a small emergency generator, which then powers the compressors needed to fire up the big ~7MW main generators. It's a delicate chain reaction — and that's just for one isolated platform.
A full grid black start is orders of magnitude more complex. You’re not just reviving one machine — you’re trying to bring back entire islands of infrastructure, synchronize them perfectly, and pray nothing trips out along the way. Watching a rig wake up is impressive. Restarting a whole country’s grid is heroic.
I remember talking to my ex's dad about his job, which involved planning refuels of a large nuclear-powered generation station in the Lower Midwest.
The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. It's a system basically built to produce "common accidents". It's amazing that it doesn't on a regular basis.
> The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true.
Funny thing is, those are the exact words I use when talking to people about networking. And realistically anytime I dig deep into the underlying details of any big enough system I walk away with that impression. At scale, I think any system is less “controlled and planned precision” and more “harnessed chaos with a lot of resiliency to the unpredictability of that chaos”
This is one of the key insights in my early SRE career that changed how I viewed software engineering at scale.
Components aren’t reliable. The whole thing might be duct tape and popsicle sticks. But the trick for SRE work is to create stability from unreliable components by isolating and routing around failures.
It’s part of what made chaos engineering so effective. From randomly slowing down disk/network speed to unplugging server racks to making entire datacenters go dark - you intentionally introduce all sorts of crazy failure modes to intentionally break things and make sure the system remains metastable.
Everything is chaos, seek not to control it or you will lose your mind.
Seek only to understand it well enough to harness the chaos for more subtle useful purpose, for from chaos comes all the beauty and life in the universe.
Message on a mug: "if carpenters built houses the way programmers write software, a woodpecker could destroy civilization."
The syncronasation of a power grid ... Wow.
If houses could be torn down and recreated with the press of a key, we probably wouldn't have a housing shortage.
We would instead have HaaS, with monthly subscriptions for a license to use the house. Which can be randomly revoked at any moment if the company doesn't feel like supporting it is profitable enough, or if an AI thinks your electricity usage is suspicious and permabans you from using a home in the entire town.
Isn't that called "renting"?
So, in short, being a tenant in a low-regulatory environment.
Tell me more about this paradise.
Ha. I live in Australia and unfortunately that’s already about the quality level for new builds.
> those are the exact words I use when talking to people about networking
Or the U.S. financial system. Or civilization in general.
It ultimately comes down to shared norms, shared expectations, and trust.
A bit of a tangent, but I don't think this is it. There are plenty of species with plenty of shared norms, expectations, and trust - but no civilization. And, vice versa, many of the greatest societies have been riddled with completely incompatible worldviews yet created amazing civilizations. Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart!
The reason people work together is fundamentally the same reason you go to work - self interest. You're rarely there because you genuinely believe in the mission or product - mostly you just want to get paid and then go do your own thing. And that's basically the gears of society in a nutshell. But you need the intelligence to understand the bigger picture of things.
For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps. They even wage war over them and in efforts to expand them or control their territory. This [1] war was something that revolutionized our understanding of primates behaviors, which had been excessively idealized beforehand. But they lack the intelligence to understand how to bring their little societies up in scale.
They understand full well how to kill the other tribe and "integrate" their females, but they never think to e.g. enslave the males, let alone higher order forms of expansion with vassalage, negotiated treaties, and so on. All of which over time trend towards where we are today, where it turns out giving somebody a little slice of your pie and letting him otherwise roam free is way more effective than just trying to dominate him.
[1] - https://en.wikipedia.org/wiki/Gombe_Chimpanzee_War
> There are plenty of species with plenty of shared norms, expectations, and trust
Citation needed on that one.
> Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart!
They spoke the same language, shared the same literature, practiced the same religion, had a long history of diplomatic ties. When the Persians razed Athens, they took refuge with the Spartans.
> For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps.
Again, I don't think this claim stands to evidence. The so called chimp war you mention is about a group of about a dozen and a huge fight that broke out among them. That doesn't support the idea that they are capable of 200-strong 'intricate' groupings.
Not the OP, but:
"They spoke the same language" ... not exactly, the Spartans spoke Doric, while the Athenians Attic. (Interestingly, there is a few Doric speakers left [0].) While those languages were related, their mutual intelligibility was limited. Instead of "Greek" as a single language, you need to treat it as a family of languages, like "Slavic".
"shared the same literature" ... famously, the Spartans weren't much into culture and art, and they left barely any written records of their own. Even the contemporaries commented on just how boring Sparta was in all regards.
If we delve deeper into ideas about how a good citizen looked like, or how law worked, the differences between Sparta and Athens are significant, if not outright massive.
While those two cities weren't entirely alien to each other, had some ties, same gods, and occassionally fought on the same side in a big war, there was indeed a huge political and cultural distance between them. I would compare it to Poland vs. Russia.
[0] https://en.wikipedia.org/wiki/Tsakonian_language
You can split linguistic hairs all you want.
Not "entirely alien, had some ties" is not it. They were part of the same cultural cluster, participated in the same games, traveled to the same sanctuaries, had mutual proxenies. The very fact that we know the opinions of several Athenians about Spartans is telling. We don't know what they thought of inhabitants of Celtic population centers, or Assyrian cities, or Egyptian ones. But we know what they thought of individual Spartans that they mention by name, biographical detail and genealogy.
I stand by my comparison to the Slavic nations of today.
Yeah, we have a lot of opinions of one another, yes we understand basic vocabulary of our cousins, though details in fine speech are another matter, yes, we are technically Christian, but still the political and societal difference between, say, Czechs and Russians is quite big.
As was the difference between the Spartans and the Athenians. Constitutionally, the poleis were all over the map, from outright tyrannies, through oligarchies and theocracies, to somewhat democratic states.
So your argument is: Athens and Sparta had things in common but were different. Like Czechia and Russia. Czechia and Russia are quite different. So were Athens and Sparta?
That's called circular reasoning.
Try to speak holistically. I have no idea what you're trying to argue. I could expand or provide evidence for everything I said, but providing a citation or proving that there are indeed social groups of upwards of 200 chimps, or whatever, isn't going to do much, because you're not really formulating any argument or contrary view yourself.
Put another way, you're arguing against an example and not a fundamental premise. Proving the example is correct doesn't really get us anywhere since presumably you disagree with the fundamental premise.
> Try to speak holistically.
That sounds very much like "Just believe me." or even more "The rules were that you guys weren’t gonna fact-check"
> I have no idea what you're trying to argue.
Presumably you know what you are trying to argue. That is what the questions were about.
> Proving the example is correct doesn't really get us anywhere
You would have solid foundations to build your premise from. That is what it would get us.
First we check the bricks (the individual facts), then we check if they were correctly built into a wall (do the arguments add up? are the conclusions supported by the reasoning and the facts?). And then we marvel at the beautiful edifice you have built from it (the premise). Going the other way around is ass-backwards.
> you're not really formulating any argument or contrary view yourself.
I don't know what viewpoint namaria has. I know that "Sparta and Athens [..] couldn't possibly have been further apart" is ahistorical. They were very similar in many regards. If you think they were that different you have watched too many modern retellings, instead of reading actual history books. That's my contrary view.
> For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps.
Here the question is what do we believe to be "societies". The researchers indeed documented hundreds of chimps visiting the same human made feeding station. Is that a society now? I don't think so, but maybe you think otherwise. What makes the Chimps' behaviour a society as opposed to just a bunch of chimps at the same place?
Which is why the long tail impact of current times is frankly terrifying.
Yes. The preppers are starting to look sane.
The preppers can only buy themselves a small amount of time, though—no more than a year or two. Eventually, their stockpiled supplies will run out, or some piece of equipment will need a replacement part.
I'd much rather focus on "prepping" by building social resiliency, instead. The local community I'm plugged into is much stronger together than anything I could possibly build individually.
For me it is the financial system as a whole.
I am an ex-scientist and an engineer and had a look at the books of my son who studies finance in the best finance school in the world (I am saying this to highlight that he will be one of the perpetrators, possibly with influence, of this mess)
The things in there are crazy. There are whole blocks that are obvious but made to sound complicated. I spent some time on a graph just to realize that they ultimately talk about solving a set of two linear equations (midfle school level).
Some pieces were not comprehensible because they did not make sense.
And then bam! A random differential equation and explanation as it was the answer to the universe. With an incorrect interpretation.
And then there are statistics that would make "sociology science" blush. Yes, they are so bad that even the, ahem, experts who do stats in sociology would be ashamed (no hate for sociology, everyone needs to eat, it is just that I was several times reviewer of thesises there and I have trauma afterwards).
The fact that finance works is because we have some kind of magical "local minimum of finance energy" from which the Trumps of this world somehow did not maybe to break from (fingers crossed) by disrupting the world too much.
I did a lot of work for a major airline earlier in my career and came away with same impression. I just couldn’t see how they kept planes in the air based on my experiences through out the organization. I think in a big enough org the sheer momentum keeps things moving despite all the fires happening constantly.
"Funny thing is, those are the exact words I use when talking to people about networking"
Computer networking is not the same. Our networks will not explode. I will grant you that they can be shite if not designed properly but they end up running slowly or not at all, but it will not combust nor explode.
If you get the basics right for ethernet then it works rather well as a massive network. You could describe it as an internetwork.
Basically, keep your layer 1 to around 200 odd maximum devices per VLAN - that works fine for IPv4. You might have to tune MAC tables for IPv6 for obvious reasons.
Your fancier switches will have some funky memory for tables of one address to other address translation eg MAC to IP n VLAN and that. That memory will be shared with other databases too, perhaps iSCSI, so you have to decide how to manage that lot.
You tried to nerdsnipe someone without mentioning L2 is effectively dead within datacenters since VXLAN became hardware accelerated in both Broadcom and "NVIDIA"(Mellanox) gear. And for those that don't need/care about L2 they don't even bother and run L3 all the way.
EVPN uses BGP to advertise MAC addresses in VXLAN networks which solves looping without magic packets, scales better and is easier to introspect.
And we didn't even get into the provider side which has been using MPLS for decades.
A problem with high bandwidth networking over fiber is that since light refracts within the fiber some light will take a longer path than other, if the widow is too short and you have too much scattering you will drop packets.
So hopefully someone doesn't bend your 100G fiber too much, if that isn't finicky idk what is, DAC cables with twinax solve it short-range for cheaper however.
If people knew how crappy, insecure, and unreliable nuclear computer systems were, there'd be a lot more existential dread about cyber security
I built control computers for nuclear reactors. Those machines are not connected to a network and are guarded by multiple stages of men with automatic machine guns. It was designed to flawlessly run 3x boards each with triple-modular-redundant processors in FPGA fabric all nine processors instruction-synced with ECC down to the Registers (including cycling the three areas of programmable fabric on the FPGAs). They cycle and test each board every month.
What’s your source?
Well, the news says that doge randos are potentially exfiltrating the details of systems like that as well as financial details of many Americans, including those who hold machine guns and probably suffer from substandard pay and bad economic prospects/job security as much as anyone else does.
Perhaps the safest assumption is that system reliability ultimately depends on quite a lot of factors that are not purely about careful engineering.
Nothing like a special commando of people doing your more malicious biddings while also being expendable
A bit off topic, but my uncle used to be security at a nuclear plant. Each year the Delta Force (his words) would conduct a surprise pentest. He said that although they were always tipped off, they never stopped them.
How is the software inspected and tested for defects, malicious or accidental? I'm just very curious about how this is done.
Almost all computers are insecure, not just the systems in nuclear stations.
Most operating systems are based on ambient authority, which is just a disaster waiting to happen.
What's the alternative?
I guess the biggest security advantage of any of these old critical systems is fact that they are not connected to the internet. At least I hope they are not.
My definition of technology is, “something built by humans that barely works”
Modern aircraft? Those are excellent and work well. I am thinking of a B787 and A350. More: How about medical implants, like a heart pacemaker?
The regulations around parts sourcing, required maintenance, and training has more to do with how well/safe modern aviation is than anything else. If those aren’t done properly, all sorts of weird things start happening. Pretty much the only reason aerospace safety records aren’t worse in third world countries is because of how obviously bad the consequences are quickly - and even then….
I love the "analog" handcranked air compressor to 7MW generator escalation, it really captures human ingenuity.
I wonder however how being part of the "continental Europe synchronous grid" affects this, and how it isolates to Portugal and Spain like this.
But yeah there are a lot of capacitors that want juice on startup that happily kills any attempt to restore power. My father had "a lot" of PA speakers at home and when we tripped the 3680w breaker (16A 220v) we had to kill some gear to get it back up again. I'm also very sure we had 230v because I lived close to the company I worked for and we ran small scale DC operations so I could monitor input voltage and frequency on SNMP so through work I had "perfect amateur" monitoring of our local grid. Just for fun I got notifications if the frequency dropped more than .1 and it happened, but rarely. Hardly ever above though since that's calibrated over time like Google handle NTP leap seconds.
I love infrastructure
I saw some ancient footage of an Me-109 fighter engine being started. A tech jumped on the wing and inserted a hand crank into a slot on the side. He threw all his might into turning it, and then after a delay the propeller started turning and coughed into life.
I realized the tech must have been winding up a flywheel, and then the pilot engaged a clutch to dump the flywheel's inertia into the engine.
The engineer in me loves the simplicity and low tech approach - a ground cart isn't needed nor is a battery charger (and batteries don't work in the cold). Perfect for a battlefield airplane.
---
I saw an exhibit of an Me-262 jet fighter engine. Looking closely at the nacelle, which was cut away a bit, I noticed it enclosed a tiny piston engine. I inferred that engine was used to start the jet engine turning. It even had a pull-start handle on it! Again, no ground cart needed.
---
I was reading about the MiG-15. American fighters used a pump to supply pressurized oxygen to the pilot. The MiG-15 just used a pressurized tank of air. It provided only for a limited time at altitude, but since the MiG-15 drank fuel like a drunkard, that was enough time anyway. Of course, if the ground crew forgot to pressurize it, the pilot was in trouble.
Again, simple and effective.
>Me-109 fighter
point of trivia: Messerschmitt, yes, but Bf-109, produced by Bayerische Flugzeugwerke.
you don't want to get your flugzeug works confused
You are correct, the official moniker is Bf-109, but the Allies referred to it as the Me-109.
BTW, since we are Birds of a Feather, I bet you'd like the movie "The Blue Max". It's really hard to find on bluray, but worth it! The flying sequences are first rate, and no cgi.
Thankfully quite easy to find on torrent. Thanks for the recommendation!
Blackstart assumes *no* power is available, period. Nothing but human muscle power. Thus the first stage is always either a human pulling a starter cord or the like, or a human building up energy in some fashion that is then dumped into the system to produce a bigger surge than is possible by direct muscle power.
And, despite the news reports, this is not a true blackstart. Some power survived.
> have to hand-crank a tiny air compressor just to start a small emergency generator
Similarly, the US Navy maintains banks of pressurized air flasks to air-start emergency diesels. Total Capacity being some multiple of the required single-start capacity
On a sub, anyway, the diesel is always started with air, not just in emergencies. Makes a cool sound as it comes up.
I understand some old radial airplane engines were started with what were essentially shotgun cartridges
They’re called Coffman engine starters [1].
Random fact: Those starters are a plot point in the 1965 film The Flight of the Phoenix, where the protagonists are trying to start a plane that’s stranded in the Sahara, but only have a small supply of starter cartridges left.
[1] https://en.wikipedia.org/wiki/Coffman_engine_starter
Love that movie!
I lived for a while on a sailboat equipped with an ancient Saab tractor engine (8 whole horsepower!). Was designed for cartridge starts in cold weather, though someone had fitted an electric starter by the time I saw it
Not just radials. The Napier Sabre H24 engine in Typhoons used cartridges as well.
> hand-crank a tiny air compressor
Is that what Dr. Sattler is doing in this scene from Jurassic Park?
https://www.youtube.com/watch?v=FoW4vXnkhJw
That would be charging up the spring to throw the breaker. High voltage breakers need to switch on (or off) very quickly, to avoid damage from arcing. It's common for them to have some kind of spring or gas piston arrangement that you pump up first to give them enough energy to do that quickly.
Nice attention to detail by the filmmakers.
No, he's winding up a spring to close the circuit breaker quicker than a human hand could, which reduces/prevents and arc from forming as the electrical contacts close.
How did you remember this scene?!
When you watched Jurassic Park in a theater in your formative years, it tended to leave an impression.
https://m.youtube.com/watch?v=WgQe68kF_8M
If you were the right age when it came out in theaters in '93 (roughly between 11-15), Jurassic Park was a huge deal. Titanic was another of those in that era (although mainly to certain females).
Yep, I can still remember the immediate after-effect seeing it for the first time in theaters when I was 7.
Hold on to your culos.
Ah shit now I want that panel for my dream house
I found a Siemens WLL2F325 on eBay for only $24k
https://www.ebay.com/itm/115854984950
I can appreciate the ability to revert to hand cranking an air compressor, yet I can't help but feel that the 99.99% of events, you'd be better served with keeping a two stroke gas engine ready to go. Air compressors tend to have parts just as or more vulnerable to environmental factors, and you get a lot more power for less elbow grease out of a two stroke.
In 99.99% of real-world scenarios, the rig would have other options to bootstrap a black start—like fully charged air tanks, backup power from a support vessel, or even emergency battery systems. The hand-cranked air compressor is really a last resort tool. We test it during commissioning to prove it could work, but in most cases, it’s never used again in the rig’s working life. It’s there for the rarest situations—like if a rig was abandoned during a hurricane, drifted off station, and someone somehow ended up back onboard without normal support. It’s a true "everything else failed" kind of backup.
Nice to see that at least in some places people are actually thinking to almost-impossible scenarios and taking them into account. I have the feeling that it's quite unlike most infrastructure development nowadays, unfortunately.
The key is the responsible party's skin in that particular game. A drilling rig is a very large, very expensive, and very lucrative man-made island. The backed-up backups have backups. Not only could it be very far away from support vessels, capable of bringing it online in every situation, every minute not in production is money thrown overboard.
Very true, although I think that economic arguments can apply to most infrastructure. What are the actual costs of a day-long nationwide blackout? I have no actual idea, but I'd not be surprised if they exceed 1 billion {EUR|USD}.
The part you are missing is ‘paid by whom’. Unlikely the power companies or regulator is going to be paying that amount here. It’s all the poor saps who didn’t have sufficient backup capacity.
There will be costs/losses by the various power companies which weren’t generating during all this of course, but also fixing this is by definition outside of their control (the grid operators are the ones responsible).
I’m sure public backlash will cause some changes of course. But the same situation in Texas didn’t result in the meaningful changes one would expect.
> Texas
That’s because there is no effective regulation of the state’s power industry. Since they’re (mostly) isolated from the national grid, they aren’t required to listen to FERC, who told them repeatedly that they should winterize their power plants. And a state-level, the regulators are all chosen by the Governor, who receives huge contributions from the energy industry, so he’s in no rush to force them to pay for improvements.
The real irony was the following summer during a heatwave, when they also experienced blackouts. Texas energy: not designed for extreme cold, not designed for extreme heat. Genius!
I miss the food in Texas, but that’s about it.
Same thing happened in south Texas last year. Years of deferred maintenance on transmission lines resulted in almost two weeks of power outages from two major storms, that could have largely been avoided. The utility provider is mostly allowed to regulate itself (while donating to the campaigns of the dominant political party), and allowed to keep excess profits/return dividends to shareholders, rather than re-invest in infrastructure. There is very little regulatory structure or checks in place to ensure the grid is being maintained. And there have essentially been no consequences, other than an apology and excuses, with an attempt to raise delivery rates even higher. As a home owner, its on me to bear the additional cost of a backup generator, because I can’t rely on the state to regulate the utility to provide the service I’m forced to pay them for.
The energy industry doesn't really benefit from the lack of requirements--they could just pass the costs along.
Rather, it's the typical Republican approach--reduce costs, never mind the safety systems.
Based on how difficult it can be to start my chain-saw, snow-blower, and motorcycle after they've sat without being run for a while, I'd not recommend a gasoline-powered engine to be the only thing on stand-by.
Air compressors in adverse environments don't hold up that well either, without basic maintenance. I've had engines run seasonally for decades. It doesn't take much for them to keep working well, though doing nothing at all is an easy way to clog up the carburator.
You have to empty the compressor of wate. Some times a lot of water. I've seen a 33g tank with 25gs of water.
Compressor pistons/screws that ingest grit/dirt, or aren’t run often enough to boil the water out of them, also tend to not last long. I used to help run a volunteer workshop with an Atlas Copco screw compressor, and it died in a few years because it wasn’t being run hard enough and the screws rusted (doh!).
It shouldn't be that bad. A little fogging oil when put away and drain all the fuel. Then a little starting fluid on the first couple start attempts. Usually they start fairly quickly if they're in decent shape. And that's just for pull starts. My electric start mower starts right up after even 5 months of not running with stabilizer in the fuel.
God bless Lucas stablier, and damn Joseph Lucas, prince of darkness. Any chance the stuff in Spain is made in England?
As an ex small engine mechanic, I'd advise against using a 2 stroke for something like that. A 4 stroke would be a better bet. Better yet would be a natural gas/propane 4 stroke, since gasoline goes stale and plugs carburetors.
Small diesels could be an option but they're harder to pull start for a given size.
> Small diesels could be an option but they're harder to pull start for a given size.
I once needed to jump-start a small marine diesel, many miles from land...
There was a small lever that cuts compression. You have to get it spinning really fast before restoring compression! It's definitely a lot of work!
EDIT - Here is a cheap modern small marine diesel [1]. The operation manual suggests that you don't have to do anything to get it spinning quickly, you just have to crank it 10 times, put away the crank handle, and then flip the compression switch. That's progress!
[1] https://www.yanmar.com/marine/product/engines/1gm10-marine-d...
Lister diesel generators are much the same - half a dozen cranks, restore compression and off they go. The hand cranking can easily break your arm if you get it wrong though.
Even gas engine pull starts have a compression release function built in. That's why you need special cylinder pressure tools to check compression on most pull starts.
I did that too and crank got stuck on flywheel. To stop engine I had to climb over the engine where now-removed stairs were since my mate was clueless. Fortunately the crank handle stayed on.
Cranks and decompression levers are gone for at least 30-40 years now tho.
Do you have any small engine mechanic books you'd recommend?
They're my kryptonite, but I accept it's mostly my ignorance.
Not a manual in particular, but this is the Bob Ross of small engine repairs' channel:
https://www.youtube.com/channel/UCr_GXW2Y56hOpGchXYNqZOQ
I learned from another mechanic, no book recommendations, sorry.
R. Bruce Radcliff Small Engines 4th Edition ISBN-13: 978-0826900333, ISBN-10: 082690033X
Yup, I would certainly use propane for a blackstart system. It stores so much better than other fuels.
I'd rather bet on the simplicity of a glorified bicycle pump than the complexity of an engine any day, but then again, I'd probably have both!
Not being at all qualified to comment (though I work for a power company), I'd think the hand crank air compressor wouldn't suffer from no spark or bad gas.
Stale gas was my guess for the plan. Maintaining an emergency system is one of those things that is easy to neglect.
If stale gas is a concern, then all of the other steps in-between zero power and your full start are also screwed.
Air compressors have more valves and gaskets that are vulnerable to oxidation, especially in salty environments, so I'd have thought the upkeep between the two, the two stroke would be easier.
But it's an emergency system, not a general operation system. Thus it's not going to be exposed to the salty environment most of the time. You could certainly put the whole thing in an airtight box.
Look at how the military builds surface-based missiles these days: it's in a factory-sealed box. Molten salt batteries so they last for decades. (You don't see molten salt in most purposes because once it's been triggered it's lifespan is in minutes or even less. They're used in applications that only need to deliver power once.)
It’s quite funny to think about.
Having good, fresh fuel on an oil rig. They need an engine that can run on crude.
Diesel will run on mostly anything if it’s running… including methane in the air intake, so you need to think quickly when presented with a generator that keeps running after cutting the fuel
Oil leaking around a turbocharger rotor seal also makes for good diesel fuel, if you define "good" as an exciting uncontrolled disassembly of the engine.
Crude oil from various wells has properties varying from ‘thick, stinky, corrosive goo’ to ‘explosive, barely liquid, bubbly mess’. Also, rigs need to be careful about ignition sources, as methane leaks can be a common emergency condition for some wells/crude.
It’s not the type of thing that using directly is economically feasible, even for emergency situations.
A power station can start a decently large generator with batteries.
Maybe there are other concerns for an oil rig.
Batteries are great when they have charge. What happens if the generator doesn't want to start the first, second, and third time? How many start attempts do you get before the batteries are dead?
The hand-pumped air compressor is the tool of last resort. You can try an engine start if there's someone there who's able to pump it. You don't have to worry about how much charge is left in your batteries or whether or not the gasoline for the 2-stroke pump engine has gone stale. It's the tool that you use as an alternative to "well, the batteries are dead too, guess we're not going to start the engine tonight... let's call the helicopters and abandon ship"
The data center where I work has large diesel generators for power cuts. They are electric (battery) start. There is no capability to start them manually. The batteries are on maintenance chargers that keep them in good condition. The generators are started and tested every two weeks.
Could the batteries be dead and the generators not start? I guess but it's very unlikely. I get that on an oil rig it might be a matter of life and death and you need some kind of manual way to bootstrap but there's not much that's more reliable than a 12V lead-acid battery and a diesel engine in good condition.
Also, the data center is probably in a city, surrounded by infrastructure that could be used if necessary. An oil rig is in the middle of an ocean, and has to rely on itself.
Lead acid batteries are not exactly what I would call reliable. They require a lot of constant maintenance to ensure that they will work when you need them and they can easily degrade in such a way that they maintain voltage and appear to be good but then fail to deliver the needed amps when you demand them. This is made much worse in cold weather. Finally, if allowed to freeze when they are moderately drained, then the accumulated water inside will freeze and drastically shorten their life span.
I think I'd take Lithium Ion batteries over lead acid for almost every conceivable use-case. They are superior in almost every way. Lighter, less likely to leak acid everywhere, better long term storage (due to a low self-discharge) and better cold weather discharge performance. The only drawback would be a slightly increased risk of fire with Lithium.
Yeah. I've watched a UPS kick in with old batteries. The steps on the power gauge ticked off every couple of seconds.
Yawn. Have you ever heard of https://en.wikipedia.org/wiki/VRLA_battery#Gel_battery Mr.Model-Tee?
I worked with a telecom provider's data center that ended up having a quad redundant diesel generator failure during the first cold snap that took the Texas grid offline a few years back. They had at three fuel supplies gel and then failed to start. The fourth, as I remember, just didn't try to fire.
It's unrealistic, and if one power station is unable to use their batteries to start their emergency generator (through the absurd incompetence you describe, or more likely through a major fire, flood or assault) the grid can be started from a different one.
Black out on a rig or ship is very different to black start of a national electricity grid.
Most vessels will experience a blackout periodically and the emergency generator start fine, normally on electric or stored air start, and then the main generators will come up fine. It's really not delicate, complex or tricky - some vessels have black outs happen very often, and those that don't will test it periodically. There will also be a procedure to do it manually should automation fail.
There are air starters on some emergency generators that need handling pumping. These will also get tested periodically.
The most complex situation during black out restoration would be manual synchronisation of generators but this is nothing compared to a black start.
Hand crank? I'd think something like an oil rig would have a propane or gasoline or diesel generator with an electric start and batteries.
The point isn't to make a system that is easy. The point is to make a system that is guaranteed to work in any remotely realistic circumstance.
In a real black start, the guys might very well grab a portable generator and just use that instead. But having the option to hand crank something rather than rely on batteries that might run flat is good.
and if the entire thing depends on it, you'll give that generator a handcrank as a backup too instead of assuming the batteries ever dying or getting flooded or whatever is entirely impossible.
bringing islands together requires one to synchronize both -- frequency and phase. It is super difficult for large generators and transmission lines. transient heat dissipation can be a real bummer.
How hard is getting each island within .1Hz of correct? The full grid doesn't have much trouble, but I don't know how much cutting things down impacts that.
And then phase will align itself a couple times a minute so what's difficult about that part?
One of the very most critical parts of a solar system is the grid tie.
They can use GPS to synchronise or back-to-back DC if available.
I can’t see an answer here: how does an air compressor start an engine?
The compressor pressurizes an air tank. When the pressure in the tank is nice and high, use the compressed air to turn a turbine connected to the crankshaft of the engine.
You can also directly feed the compressed air into a cylinder (or even the intake manifold!) to force the engine to turn. No extra turbine required, though the plumbing might get a little odd. [https://en.m.wikipedia.org/wiki/Air-start_system]
That tends to be for very large engines, where the extra plumbing isn’t a problem.
This technology of starting a diesel engine using a turbine driven by compressed air was used in Russian T-34 tank during the WWII. While Germans could not start the tanks in the cold of winter 1941 from the frozen batteries the Russians were using compressed air (hand-crank) to start T-34s just fine.
It doesn't. It's a power storage device to allow muscle power to produce a higher peak output.
ChatGPT's tone is slowly taking over the entire internet
[flagged]
The fewer resources we dedicate to grid resilience and modernization, the harder black starts become. And as grids get more complex and interdependent, recovering from total failure becomes exponentially harder.
A rare but sobering opportunity to reflect on something we usually take for granted: electricity.
We live in societies where everything depends on the grid — from logistics and healthcare to communications and financial systems. And yet, public awareness of the infrastructure behind it is shockingly low. We tend to notice the power grid only when it breaks.
We’ve neglected it for decades. In many regions, burying power lines is dismissed as “too expensive.” But compare that cost to the consequences of grid collapse in extreme weather, cyberattacks, or even solar storms — the stakes are existential. High-impact, low-frequency events are easy to ignore until they’re not.
Just to highlight this: the last significant power outage in Western/middle Europe was 2003. [1]
That's 20 years without any significant problems in the grid, apart from small localized outages.
It's not hard to start taking things for granted if it works perfectly for 20 years.
Many people don't even have cash anymore, either in their wallet or at home. In case of a longer power outage a significant part of the population might not even be able to buy food for days.
[1] https://en.wikipedia.org/wiki/2003_Italy_blackout
> Many people don't even have cash anymore, either in their wallet or at home.
Even if you have cash many shops would not sell anything in case of a mass outage because registers are just clients which depend on a cloud to register a transaction. Not reliable but cheap when it works.
Many supermarket chains (in the West at least) have satellite links at their major locations because they can't afford to close a store just because the local ISP had an issue.
The real question is how long can some of the smaller banks' datacenters stay up.
Firstly and most importantly, a cash register needs a power outlet. It is highly contestable that every single Western supermarket out there has a diesel generator down in the back / storage room that will kick in in an instant if a power outage begins.
Lest also forget the Crowdstrike drama where many supermarkets simply went dark, in some instances for nearly 24 hours, despite working communication links. But I digress.
Crowdstrike was an interesting one. Just as it was going down I went out to the supermarket and found half of the self checkouts had bluescreened. Then a few hours later they were all back and functioning again. The supermarket had remote management at a level below the OS that could restore the whole countries self checkouts rapidly.
I would not be surprised if they simply booted an image from the network. It would significantly simplify maintenance, as for any change you'd just need to update a single image and push it downstream to an in-store management server. The individual terminals essentially become disposable.
> It is highly contestable that every single Western supermarket out there has a diesel generator down in the back / storage room that will kick in in an instant if a power outage begins.
Literally true. However:
- If it takes them 10 minutes to fire up the generator, then 5 minutes to restart the network and registers, that is no big issue (in a many-hour outage)
- At least in my part of the USA, many supermarkets do have generators - because storm damage causes local outages relatively often, and they'd lose a lot of money if they couldn't keep their freezers and refrigerators powered. Since the power requirements of the lighting and registers are just (compared to the cooling equipment) a rounding error, those are also on generators.
Plus, there are backup-power lories and refrigerated trailers. If your shop doesn't have enough backup power for duration, you might see several of these pull into the carpark all at once. If not all of the chillers can be powered, shop's staff will schlep stuff to the refrigerated lories.
Seen it done in USA, for a Target next to a Kroger grocer. Kroger lost everything that needed cold after reserve either ran out or wouldn't start, but Target had a contingencies contract and lost no product.
Well,in my experience it was the case for the 2 largest supermarket chains. We lost electricity at 12:30pm and only got it back during the night at 3am.
But both major supermarkets nearby worked on diesel generators and payment by card worked flawlessly. I guess they had satellite connection.
It might have been more complicated in small villages but people living in rural areas ually still use a lot of cash.
In my local area of Spain/portugal, 2/3 supermarkets and 2/3 gas stations had generators up and running within a couple of hours. We’re pretty rural, though - I don’t know that urban areas faired as well
We're outside Mataro, had to make a trip into Barcelona yesterday. I'd say most gas stations north of Barcelona/Maresme area were 100% offline, some (we found only two, from 6 visited) gas stations still had operational pumps but huge queues and cash only. None of the TPVs seemed to work anywhere in the afternoon yesterday here, even the battery powered/mobile network ones.
That's true, I went shopping cca. 4-5 hours after the blackout started and had no issues, even card transactions worked. Whoever designed the retailer, they clearly had this scenario in mind. Even the "self-service" computer kiosks all worked.
In a multi-day event like we are talking about here, couldn't a shop owner revert to a paper ledger? I mean, it would suck and transactions would take much longer but if the alternative is people starving or having your inventory looted by a desperate mob, a nineteenth century solution seems preferable.
They can and do. They will also do deep cuts on prices/give away for free for refrigerated and frozen goods because those will just get tossed and neighborhood goodwill is still a thing in some places.
There was a 4-day power outage here (Seattle suburbs) last fall, and one of the auto parts store made an effort to serve customers even though they didn't have any power. I paid cash, and I forget whether or how they did credit card transactions (possibly by writing down CC numbers on paper.) They made a lot of phone calls to a different store to get prices for items.
I'm located in Barcelona, and yesterday lot of transactions on mini markets / pharmacies were not possible because the item prices were unknown, adding to the fact there was no phone lines available to reach out.
And in many places in Europe, registers are literally required by the government to be this way, for VAT fraud avoidance reasons.
It's probably just Hungary though, I don't know of any other country mandating an always-online cash register.
Portugal has mandatory electronic receipts. By this I don't mean email receipts, I mean that all receipts have a code on them that is then also available to be looked up on the government's side (e-Fatura is the app for this) for tax reasons. I think it's fairly simple though, just registers the total amount, the seller, and how much VAT was paid.
However I assume this can work offline with the data being uploaded later though, as basically all the small supermarkets and shops were still open here (_incredibly_ chaotic though), and on the big supermarkets card payments were working (TBF, even the free wifi was working there, I guess they probably have some satellite connection).
It's an EU regulation.
> Many people don't even have cash anymore, either in their wallet or at home. In case of a longer power outage a significant part of the population might not even be able to buy food for days.
So, what's really interesting is that these sorts of social collapses have happened. In fact, they often happen when natural disasters strike.
When they do happen, mutual aid networks just sort of naturally spring up and capitalism ends up taking a backseat. All the sudden worrying about the profits of Walmart are far less important than making sure those around you don't starve.
As it turns out, most people, even managers of stores, aren't so heartless as to let huge portions of the population starve. Everyone expects "mad max" but that scenario simply hasn't played out in any natural disaster. In fact, it mostly only ends up being like that when central authority arrives and starts to try and bring "order" back.
You can read about this behavior in "A Paradise Built in Hell" [1].
[1] https://en.wikipedia.org/wiki/A_Paradise_Built_in_Hell
Well if the situation happens that people can't buy food, things will easily become nasty quickly. People will break open stores and use violence to get what they need. So cash money won't really help a lot in a serious situation.
The "civism" was well noted in the Spanish case, as far as I know, the whole country passed this incident without a single case of security issue. During the whole thing, people were extremely kind and polite. People sitting in dark bars was kind of "funny" how people clinging to their normal lives even in emergencies.
Shop owners would just take notes on who buy what and let people go with the items.
Looting only ever happens when areas hve started being evacuated and most shop owners + law enforcement are elsewhere.
And then take it up the butt from the taxman once he finds out.
But but but, muh crypto is the future </sarcasm>
I often wonder if we should leave energy/telecommunications in a state where they can and do fail with some degree of frequency that reminds us to have a back up plan that works.
I had thought that the (relatively) recent lockdowns had taught us how fragile our systems are, and that people need a local cache of shelf stable foods, currency, and community (who else discovered that they had neighbours during that time!)
For something like this, a local electricity generation system (solar panels, wind/water turbines, or even a ICE generator) would go a long way to ensuring people continued to have electricity for important things (freezers)
There are now ubiquitous wireless POS terminals for card payments that can be recharged from emergency sources of electricity(like cars). As long as the mobile internet works it's possible. Of course this only little alleviates the disruption.
As long as the mobile internet works it's possible.
If we're talking about a situation where the grid goes down, the mobile internet is most definitely not working.
Surprisingly mobile networks seemed to stay up in Portugal. I'm not sure to what extent and if they lasted for the whole duration of the blackout. They definitely limited consumer use though.
Most mobile base stations have a limited backup battery and some have generators on site. I'd expect telephone infrastructure to have 24-48 hours of backup in the USA and I don't know why Europe would be much different.
Population density is pretty high in many European inner cities. Most of the cell sites around here are on top of apartment buildings and I doubt they have a genset. Here in central BCN the mobile network was completely offline within an hour or two of the power going out.
I love in Spain and in my zone I didn't have Internet access until after 22:00, and it was sloppy at best.
0:49 and light came back, and woke us for a moment.
So yeah, you need local first POS applications.
Surely there are still card POS systems that can buffer transactions? Sure you loose some part of the system like payment authorizations but the potential loss of money is lower than closing shop.
> a significant part of the population might not even be able to buy food for days.
And who's fault is that? Why did europe allow this?
Why will the US allow this, eventually?
> And who's fault is that? Why did europe allow this?
In Spain it's now illegal to pay with cash for transactions over 1000EUR. Absurd.
In Norway they recently made it mandatory in most circumstances to accept cash for transactions up to 20,000kroner (~1700EUR): https://www.norges-bank.no/en/topics/notes-and-coins/the-rig...
The 1000EUR limit doesn't apply between private individuals; for businesses, you will find many other European countries also don't take large cash payments, for security or convenience reasons. E.g. you can't buy a car from a dealership for cash in "cash is king" Germany either. They expect a wire in almost all cases.
I don't think anyone is going to starve to death because they can only spend 1000EUR...
I don't know but I find it very practical to not carry piles of cash in my pocket and home and know that we're less likely to get robbed just because of the cash we have.
I don't know how true the relationship between the cashless lifestyle and safety actually is, but it works and I feel ok; I'm not sure that the prospect of a few hours of national blackout once in 20 years will make me change my mind significantly.
Today I was able to walk into a grocery store, pay for food, and go home to have a warm lunch (having a gas stove also helped tremendously). The matter was having a 10€ note at home. Not what I'd call "piles of cash".
As an added benefit, no bank knows where I bought and when, which I find is a great advantage over the alternatives. (I also use Gpay; this comes from someone who just found a good middle point without forgetting about the more reliable, physical and privacy friendly option)
I think I have 10€ laying around at all times, possibly in loose change between my home and car. I do not always walk around with that in my pocket and I never have more than 500€ at home.
I didn't mean literally zero cash, but once the bulk of your transactions are by card, you don't need to constantly go to the ATM and replenish your cash reserves
And I myself didn't mean that I happened to have a measly 10€ note at home, I normally have around 300 minimum, to spend organically on purchases.
Of course I get that carrying coins and notes is cumbersome, but if we've managed to live all through the 80's and 90's with it, I think we can manage to keep doing it. 100% digital money is giving up on a huge level of self-determination and privacy that I wouldn't feel comfortable with, but I guess as newer generations grow up already pre-adoctrinated and not being able to compare the before-and-after, in the end society will end giving up.
I got my first cell phone as a full bearded adult so I do remember the times where you carried cash and the time you could actually meet people in places without having to constantly update one's position.
I don't think it's just a generational divide.
I do understand the privacy and self-determination problems of a cashless society but I have to admit I'm just to weak-minded to care about that in practice; the practicality of just paying even for just coffee with my phone is just too big for me to care for it.
> the time you could actually meet people in places without having to constantly update one's position
Not sure I understand how that's different than today? You set a time and place, then you meet there, are people doing more than that today? Seems the youngsters understand this concept as well as older people, at least from the people I tend to meet like that.
> the practicality of just paying even for just coffee with my phone is just too big for me to care for it.
Interestingly enough, no matter if you had cash or card yesterday you couldn't get a coffee anywhere, as none of the coffee machines had power and even in the fancier places where they could have made the coffee without power, they didn't have electricity for the grinder itself, so no coffee even for them.
> You set a time and place, then you meet there, are people doing more than that today?
no, today people are continuously updating you about their whereabouts and assume you can just change time, place continuously and if you don't have the phone people get lost and panic. Ok I'm exaggerating of course, but there is a grain of truth in this
It does seem safer to not carry cash. However, I remember around May 2024 that there were some reported incidents in Chicago were in the early morning hours, there were some groups of robbers who would force victims to do something I found especially worrying: they would be forced into resetting their phone password and logging into their mobile banking app. I’m not sure what ended up happening in these cases.
Cash is extremely easy to conceal. fear of burglary is a ridiculous reason not to have some saved for a rainy day
More or less.
I once used an aggregator app to summon a handyman to my place. My request was simple: move two pieces of furniture around my very small apartment.
So I find a reputable service within the app, I schedule it, and they send a guy. He shows up to my door breathless, with some kind of sob story about a vehicle breakdown. I dismiss that out of hand and he gets to work. He did a fine job and it didn't take very long.
Then we get to the point of settling up, so I announce I'm going to pay in the app. He looks really disappointed and says he usually takes cash. I realized at this point that he was ready to shake me down, and also he would incidentally be discovering where I stashed my cash, when I reached for it with him there in the room. So disappointed. So I send the money out in the app and I show the confirmation screen to the guy. And I felt so bad that I followed up with a tip in the same fashion.
But at the end of the day he was just a garden-variety cash-in-hand scammer and I had no reason to feel guilty, because I had unwittingly outwitted him by trusting the app. And the company had no qualms about it.
Another time, I had a very short cab ride to the laundry. And it did not take long for the driver to spin a gigantic tale about his auntie addicted to gambling used up all their savings and they was really hurting for money. I was shifting uncomfortably wondering why I was hearing this. So the cabbie parks the car and his POS machine shuts off. He's like "oh it's out of order" so here he is, shaking me down and expecting me to go fetch cash to put in his grubby hands.
I stared him up and down, started taking photos, and got out of there. I discussed with dispatch. They said if he's not accepting cards and I intended to pay by card, I owe him nothing.
So again a cash-in-hand sob-story scammer was foiled. The cab service was crazy enough to assign him to pick me up additional times. This is why I ride Waymo, folks!!!
This might be one of the most paranoid things I've ever read on HN.
The laborer was simply trying to actually get paid vs. deal with the overhead of the app. Somewhat shady perhaps - since it routes around the company taking their cut for finding him the work, and likely avoids taxes. I've paid these sorts of guys cash every single time I've used such a service and exactly zero of them have "shook me down" or cared where I stored the money. They make so little already I'm happy to help them out with a smile.
Cabbies simply want cash for pretty much the same reason. They get charged an astronomical "service fee" by the cab company, and likely are avoiding taxes as well. I agree that such a situation is more shady in general, but I've actually had (NYC) cops side with cabbies on this topic and force me to go get cash at the ATM or get arrested. I also use car services now over cabs whenever possible due to this reason - mostly for convenience, never out of fear of being robbed though.
The chances of you getting mugged/stolen from for using cash are just the same as the chances of you getting mugged for no apparent reason walking home. Perhaps the collective dis-use of cash has reduced these odds, but you specifically is utterly irrelevant.
Lol what? How do you know either of these guys were trying to rob you? They prefer cash because they can take it under the table and not report it for taxes. Everybody knows that.
> take it under the table
Yes, well, I choose not to participate in shady shit like that. Is that OK that I prefer to make transactions as laid out by their employer and not every random guy?
> happy to help them out with a smile
So you choose to be knowingly complicit in tax-avoidance schemes. That's fine; you do you, but some of us steer clear of shady shit, just on principle, you know? Perhaps the company deserves their cut as well -- they get paid so little already, amirite?
Also if there was nothing unusual about their choice of payment, then why must they regale me with these shitty sob stories? Am I supposed to be moved to tears at their hardship and heroism at making it to my door, that I must promptly cover their expenses? They are not panhandlers, they are service providers.
No, I ordered a service and I pay for the value of the service, according to the Company's rates. The cab company was clear about it: either I pay how I want to or I don't owe them. Nobody's arresting me for refusing to fork over cash. That's a scam.
That's not the only time I was cash-scammed by a cab driver. They will pull every trick in the book, and surely they compare and trade notes on their marks.
It gets even worse: my simple insistence on transacting with the cab company earned me fake receipts. Yes, they faked every receipt that they sent me in email. The totals were all fudged down to be much smaller than what I paid, including a $0 tip. It was very very obvious, especially when the rides booked in the app were generating duplicates showing different calculations. I reported it twice to their backend developers and they said that there were some coding errors in device drivers; please stand by for a fix. LOL!
That's a scam to ensure that taxpayers can't get reimbursed for out-of-pocket medical expenses. Most/all cab companies provide NEMT services as well, and they can't stand when people go outside of insurance companies. So they falsified my receipts.
And again, that's why I trust Waymo.
[flagged]
Why will the US allow this? Because there's too much money to be made by middle men who want their cut of the transactions.
We will use better technology for electronic transactions. Most of banks worldwide still use COBOL for most (all?) their software infrastructure.
You can do as many electronic transactions as you wish without internet or electricity, provided you have something with charged battery. Problem is the transaction cannot be verified without internet, but when internet gets restored, all transactions can be applied.
That technology exists for more than a decade, so banks will implement it in 20 or 50 years. Most sane people will not wait patiently for half a century till some software engineer implements electronic transactions with COBOL, and we will use some kind of blockchain much sooner than that.
I see no problem with COBOL. I'd rather have my credit card transaction be processed with 40yo well tested COBOL than Java from the newest intern using copilot.
> Most of banks worldwide still use COBOL for most (all?) their software infrastructure.
Nahhh - some banks have some parts of the infrastructure in COBOL. Specifically larger retail banks often have their ledgers in COBOL. Most of them want rid and are actively getting rid. Most places have had programs to root COBOL out since before 2000, but there are residual implementations remain. The ledgers are the hardest place to deal with because of the business case as well as the awkwardness. Basically there's not much of an advantage (or at least hasn't been) in modernizing so keeping the thing going has been the option. Now people want to have more flexible core systems so that they can offer more products, although not so sure that customers want this or can consume it. Still - it supports the idea of modernisation so not many people are keen to challenge.
The most common big implementations I come across are in Java.
They use Go as well, I know someone who writes it for banks, but most of their infrastructure is written in COBOL. There are some sources [1], and some people have told me the same in person, not in exact numbers or percentages, but roughly the same.
Anyway point remains, electronic transactions with no internet or electricity is a solved problem, and banks don't want to solve it or they can't due to incompetency or maliciousness.
Currency transactions worth their weight in gold, it is of utmost importance for transactions to always be published to a central authority right away. If they don't have to be published, they should not exist at all. Imagine people buying stuff without anyone knowing right away! That should never, ever exist, for any reason.
[1] https://thenextweb.com/news/ancient-programming-language-cob...
It’s not incompetence it’s regulations and inertia. Getting your central banking system past the regulators in most countries is difficult enough that it shades the effort required to replace the core systems.
> Imagine people buying stuff without anyone knowing right away! That should never, ever exist, for any reason.
I can imagine it and it happens all the time. Your version sounds very dystopian.
Anti tax evasion and money laundering measures
Yeah, this is the turkey’s dilemma - life on a farm is a lot better than life in the wild for 51 out of the 52 weeks of the year.
Most of our modern economy and systems are built to reduce redundancy and buffers - ever since the era of “just in time” manufacturing, we’ve done our best to strip out any “fat” from our systems to reduce costs. Consequently, any time we face anything but the most idealized conditions, the whole system collapses.
The problem is that, culturally, we’re extremely short-termist- normally I’d take this occasion to dunk on MBAs, and they deserve it, but broadly as a people we’re bad at recognizing just how far down the road you need to kick a can so you’re not the one who has to deal with it next time and we’ve gotten pretty lazy about actually doing the work required to build something durable.
"Just in time" is a phrase I hate with vehement passion. You aren't optimizing the system, you're reducing safety marigns - and consequences are usually similar to Challanger.
This is a solution that teenager put in management position would think of(along with hire more people as solution to inefficient processes), not a paid professional.
Systems like electric grid, internal water management (anti-flood) shouldn't be lean, they should be antifragile.
What's even more annoying that we have solutions for a lot of those problems - in case of electric grids we have hydroelectric buffers, we have types of powerplants that are easier to shutdown and startup than coal, gas or wind/solar(which cannot be used for cold start at all).
The problem is that building any of this takes longer than one political term.
> You aren't optimizing the system
Of course not, they're optimising shareholder profit.
> Systems like electric grid, internal water management (anti-flood) shouldn't be lean, they should be antifragile.
How do you make those systems antifragile rather than simply highly resilient?
Things which can’t self improve can’t be antifragile by definition. NNT alludes to this multiple times - systems together with processes and people running them can be antifragile, but just things cannot.
I postulate the grid as a whole is antifragile, but not enough for the renewable era. We still don’t know what was the root cause of the Spanish blackout almost 24h after it happened.
Step 1: read a book by nnt
Yeah, I have, that’s why I’m asking the question.
JIT isn’t about reducing safety margin. It was pioneered by Japanese companies, namely Toyota. They are known for risk adverse, safety first.
> This is a solution that teenager put in management position would think of(along with hire more people as solution to inefficient processes), not a paid professional.
What kind of comment is this? Toyota has been using and refining it for decades. It wasn’t invented yesterday by some “teenagers”. Such a state of HN’s comment section.
JIT is definitely not perfect as exposed during the Covid period, but it isn’t without merits and its goal isn’t “reducing safety margin”.
Then we have JIT in computing, such as JVM.
> its goal isn’t “reducing safety margin”
Sure it is. That's exactly how it achieves the higher profitability. Safety margin costs money. Otherwise known as inefficiency.
Slack in the system is a good thing, not a bad thing. Operating at 95% capacity 24x7 is a horrible idea for society in general. It means you can't "burst mode" for a short period of time during a true emergency.
It's basically ignoring long tail risk to chase near-term profits. It's a whole lot of otherwise smart people optimizing for local maxima while ignoring the big picture. Certainly understandable given our economic and social systems, but still catastrophic in the end one day.
It literally is reducing safety margin(buffers) of a whole distribution system by definition, and it is also being applied in places where it does not fit - like systems that should be resilient to disruption and/or anti-fragile.
I would expect a paid professional in management discipline to be aware of such nuance but alas proven wrong again.
Challenger wasn't really about cutting safety margins, but about kicking the can on a known problem: blowby in the motor joints. It was a gut feeling by the engineers that the problem was related to temperature, but there was enough of a random element to it that there was nothing specific to point to.
That should have been enough to scrub anyway, but there was clearly political pressure to launch.
I do agree that they need to specifically design anti-fragile.
What are some examples of modern system collapse?
We've had substantial disruptions, but they've not been particularly irrecoverable or sustained.
For the people who died of normally preventable death during covid while the health services were overwhelmed, the damage is irrecoverable. The chips shortage lasted years. Every year we become more, not less, dependent on the supply chain working. Every year we become less, not more, resilient.
I don't think it's crass to separate the deaths that occurred from a novel disease from the impact it had in society. In the medium term, it's a blip, never mind the long term. There's a huge chunk of society that thinks there was a huge overreaction!
The chips shortage has been difficult, but it's also been little more than an inconvenience when you look at it in terms of goods being available to consumers or whatever.
That chunk is heavily influenced by the propaganda that over a million dead people isn't a big deal. The propaganda is economically incentivized because slowing down the economy is bad for business even if it protects human lives.
Do you consider this significant failure a collapse?
I fell into the other poster's trap, talking about something emotionally charged that isn't really responsive to what I said initially.
Nearly all collapses are of limited temporal duration (except for extinction events....). I think it is fair to call a health system that failed to protect the nation and world a collapse. It failed to perform its function in a dramatic way. Now, the fact that most of us survived at least is being exploited to say it was no big deal and actually, why not trash every public health institution so the economy is never shut down again?
Sad whomp whomp horn: the economy is going to be negatively affected by covid disability and death on an ongoing basis and a new pandemic will still cause so much fear the economy will shut down.
We have plenty of small scale collapses that weren't of limited duration. It's just that such things are typically only noted by archeologists. We only see the survivors and thus conclude that collapse isn't an existential problem.
I do agree on Covid disability. Early on we saw some pretty dire predictions, but since then it's mostly been an exercise in muddying the waters. Lots of wheel-spinning about what constitutes long Covid when they should have simply been collecting data on the various symptoms. Better to not see the problem than have to deal with it.
Look at how we were handling AIDS before we discovered it was HIV destroying the immune system. Long Covid is still at that stage--we are seeing a slew of highly varied effects rather than the mechanism.
The post you are replying to is not talking about the Covid deaths, but rather about the deaths from other causes triggered by the Covid disruptions. When a trauma case dies of the lack of a ventilator because they're all in use on Covid patients. When the trauma case bleeds out at the scene because the ambulance is running a Covid patient to the hospital.
And a lot of people thinking it was an overreaction proves nothing. People don't get a vote on reality.
[dead]
Honest question, are we better off in the long run, and is it a better solution, to decentralize energy generation and make more smaller grids rather than linking them all up? This isn't to say completely getting rid of the ability to transfer between the smaller grids to assist with power disruptions but to decouple and make it less likely for catastrophic "global" failures like this.
With a high fraction of renewables, the reverse is probably better in the long run. The larger geographic area you connect, the less you're affected by weather systems, and the wider area you can draw dependable dispatchable power such as hydro from. But that depends on having enough grid capacity to move enough power around, which is currently a problem.
But I wonder from a reliability (or lack of cascading failures) point of view whether synchronous islands interconnected with DC interconnects is more robust than a large synchronous network?
It's hard to build big generators, so as we already struggle with infrastructure I don't think that's feasible, but it would be great.
We're slowly reaching this point with the internet too.
I feel like to many technologists, the internet is still "the place you go to to play games and chat with friends", just like it was 20 years ago. Even if our brains know it isn't true, our hearts still feel that way.
I sometimes feel like the countries cutting off internet access during high school final exams have a point. If you know the internet will be off and on a few days a year, your systems will be designed accordingly, and if anything breaks, you'll notice quickly and during a low-stakes situation.
Burying lines is not a panacea, it generates massive reactance changes compared to a classic line.
Maybe a good reason (in parts of the world where this is practical) to have some solar + battery storage. Doesn't even need to fully replace grid power, just enough to run the barebones when the grid goes out.
If we are talking about a 72 hours blackout, I would be concerned about water and sewer working before one even thinks about home electricity.
Houses have water tank that work as buffer for when the water stop, couldn't residential batteries work the same way?, it could detect drops in voltage and stop charging, so even if the grid is unstable in a black out, the residential isn't going to light up immediately, or only the houses without batteries
Interestingly it seems that the black start drill is considering a smaller zone of impact than what has happened here.
Also I suspect there is far more renewables on the grid now than in 2016.
This is potentially the first real black start of a grid with high renewable (solar/wind) penetration that I am aware of. Black starts with grids like this I imagine are much more technically challenging because you have generation coming on the grid (or not coming on) that you don't expect and you have to hope all the equipment is working correctly on "(semi)-distributed" generation assets which probably don't have the same level of technical oversight that a major gas/coal/nuclear/hydro plant does.
I put in another comment about the 2019 outage which was happened because a trip on a 400kV line caused a giant offshore wind farm to trip because its voltage regulator detected a problem it shouldn't have tripped the entire wind output over.
Eg: if you are doing a black start and then suddenly a bunch of smallish ~10MW solar farms start producing and feeding back in "automatically", you could then cause another trip because there isn't enough load for that. Same with rooftop solar.
This is potentially the first real black start of a grid with high renewable (solar/wind) penetration that I am aware of.
The South Australia System Black in 2016 would count - SA already had high wind and rooftop solar penetration back then. There's a detailed report here if you're interested:
https://www.aemo.com.au/-/media/Files/Electricity/NEM/Market...
Grid tied solar won't put power into the grid when the grid is down. It's the one reason I didn't grid tie.
Non tied solar won't affect the grid at all. So this is a non-issue.
Grid tie requires the grid to tie to, otherwise it can't synchronize. So it stays disconnected.
"Grid tied solar won't put power into the grid when the grid is down. It's the one reason I didn't grid tie."
Why would that prevent you from being grid-tie? I have 53 panels (~21kw) grid tied and pushing to the grid, but in the event of grid failure my panels will still operate and push into my 42kwh battery array which will power the entire house. ( The batteries take over as the 'virtual grid source). I can then augment the batteries with generator and run fully off grid for an extended amount of time ( weeks in my case ).
> grid tied
> 42kwh battery array
these are mutually exclusive. What you have is a hybrid system, which is something i explicitly did not mention. A grid-tie system is not generating an AC waveform when the grid is down, at all. it cannot, by definition, and by design, as the AC waveform requires the grid to synchronize to.
Hope this clears this up.
Youre missing my point. I know that; I mean if you are restarting the grid and say you have a segment you think has 5MW of load that will come online, but you connect it and a minute or two later suddenly 2MW of grid tied solar detects the grid and starts exporting and you now have 3MW of load it is going to make it much more tricky to balance the restart. I'm not sure how much of a problem this is in reality but it seems to me restarting a grid is made much more tricky when you have millions of generation assets with no control over.
This is why most of the restart of the grid is being done as the solar input tappers out I suspect. The grid was pretty much down during peak sun and started coming back online around 5:30 pm or later.
> I mean if you are restarting the grid and say you have a segment you think has 5MW of load that will come online, but you connect it and a minute or two later suddenly 2MW of grid tied solar detects the grid and starts exporting and you now have 3MW of load it is going to make it much more tricky to balance the restart.
I really thought that sentence was going to end with "it makes it a lot easier to handle that segment".
Yeah you have some big problems if it's a complete surprise, but your status quo monitoring would have to be very strangely broken for it to be a complete surprise. Instead it should be a mild complicating factor while also being something that reduces your load a lot and lets you get things running quicker.
Those small solar panels probably go directly to the next consumer.
You need to calculate for it but I don't think this would be a problem
Practical Engineering did a really great video a few years ago on why black starts are hard, complete with a tabletop demo about the physics of synchronizing large spinning generators: https://www.youtube.com/watch?v=uOSnQM1Zu4w
Spain is part of the largest interconnected grid in the world: https://en.wikipedia.org/wiki/Continental_Europe_Synchronous...
Does this really qualify as "black start" when they can rely on the bigger EU grid?
Yes because they have to bring it all back up in phases so that they only face the load spike* from one interconnect at a time, which can take some time and can fail if there’s unknown damage like the GP said.
It really depends on the region though because almost all large hydroelectric dams are designed to be primary black-start sources to restore interconnects and get other power plants back up quickly in phase with the dam. i.e. in the US 40% of the country has them so it’s relatively easy to do. The hardest part is usually the messy human coordination bit because none of this stuff is automated (or possible even automatable).
* the load spike from everyone’s motors and compressors booting up at the same time
Presumably emergency phone comms still work(?) so they could issue instructions to do a phased (heh!) restart to avoid every fridge/air conditioner/whatever restarting at once. Not sure how successful that would be however.
There’s usually two parallel processes going on during a black-start: spinning the power plants back up to get them synchronized to the grid, and getting power back out to all the consumers. Power plants have breakers that disconnect them from consumers which they keep disconnected until independent connections to other power plants allow them to spin up their massive turbines and synchronize them to the grid. There’s also tons of other downstream breakers at substations which will be in various states of functionality.
The power plants with direct connections have hard lines and black-start procedures that get power out to the most important customers like telecom infrastructure, which provides the rest of the comms. In a real world full restart it’s going to mean organizing workers at many substations to babysit old infrastructure so cellular is pretty much mandatory.
And to add more fun, this time they're not dealing with a small number of individual power plants that can be connected with only some phone-call based coordination.
Instead, there are literally hundreds of smaller wind/solar installations. Some of which depend on rapidly fading cellular communication to restart. And some might need an actual site visit to throw on the physical breakers.
Spain is but Portugal is only connected to Spain and they are currently doing a full black start.
For Spain the external power and synchronization can come from France rather than generators which will help, but the process and complexities are still mostly the same. Call it a dark start, perhaps.
Even then, the interconnect capacity with France is currently 2800 MW (was 1400 MW in 2015) with just 3 links, all on the NE.
If Portugal (on the West) had to wait for that, it would probably have taken even longer.
But even a "small" amount of input power makes the process much easier because it can be used to bring up the next source of power.
Sure. Portugal is in deed completely “isolated” if it loses Spain, so it needs black start capability if it wants some degree of autonomy.
As far as containing the issue, this was a disaster. On the flip side, this was as good an opportunity to test a black start as any, it went reasonably well, and the network operator was already in the process of contracting two further dams for the ability.
Presumably not
> A black start is the process of restoring an electric power station, a part of an electric grid or an industrial plant, to operation without relying on the external electric power transmission network to recover from a total or partial shutdown.[1]
https://en.wikipedia.org/wiki/Black_start
The key part of the phrase is actually "electric power station, a part of an electric grid or an industrial plant." Note how the definition doesn't include an entire grid.
Only the first power plant in a black-start (like a hydroelectric dam or gas plant started by a backup generator) is truly "black started." The rest don't fit that definition because they depend on an external power source to spin up and synchronize frequency before burning fuel and supplying any energy to the grid. If they didn't, the second they'd turn on they'd experience catastrophic unscheduled disassembly of the (very big) turbines.
Only the first power plant can come online without the external transmission network.
That’s not necessarily 100 true, as you could have multiple plants black start and then synchronize later.
In fact, if you’re not sure which will start first, you might go that way. They’re all disconnected from the grid at that time anyway.
You’re absolutely right, there are a lot of variations. In this case I think Portugal started from their own hydroelectric dam and restarted everything North to South while Spain started several in parallel from the interconnects to the rest of Europe (can’t tell which interconnect it was though).
Yup - a true black start would be substantially more difficult and time consuming. More like a gradual reconnect.
Then again they might be less prepared precise because of the euro grid is available
It's a complex enough issue that you can't really 'hand wavey' it. The details are specific and thorny.
But not a black start.
The frequency aspect of a black start is presumably a bit easier in Europe because there's an interconnected synchronous grid so they can bootstrap it from France essentially.
It's far more problematic for the UK because all the interconnects are DC.
I was recently told by an electrical engineering lecturer that the black start plan here in Ireland is to use the DC interconnectors with the UK to provide startup power to a synchronous generator.
With the new wexford-wales interconnect that went live last month, and another one planned from Cork (?) to France things might be even easier in the near future I reckon.
Did he mention the UK's plan in case they have a full blackout?
https://www.nationalgrid.com/sites/default/files/documents/H... (2017, probably current)
"it is prudent that the cost of maintaining Black Start Capability should demonstrate value to the end consumer"
That sounds like someone explaining why the solution is so bad, before describing what the solution is.
To me it sounds like an energy company attempting to excuse lack of spending on infrastructure whilst paying out millions to C-suite in bonuses and millions more to shareholders whilst arguing prices have to rise because they can't afford to spend on infrastructure...
Your incentive "analysis" is wildly incorrect.
Electricity markets and electricity networks are designed by the regulator.
Incentives are planned by the regulator so that individual stations or companies have the correct incentives to have capabilities that the network grid needs.
One example is financial incentives to provide black start capabilities. Another example is incentives to provide power during peak loading (peaker plants). There are many more examples of incentives designed so that the needs of the whole network are met.
If an operator is incentivised to act selfishly in such a way that the grid will fail, then that is a failure of the regulator (not the individual operator).
Blaming individual people or companies for systemic faults is generally a bad thinking habit to form. There are too many examples where I see individuals get blamed. Fixing our systems is hard but casting blame in the wrong places is not helpful. It's difficult to find a good balance between an individual's responsibilities and society's responsibilities.
> Electricity markets and electricity networks are designed by the regulator.
Not quite. They are _influenced_ by the regulators.
And Europe has been incentivizing trash-tier low-quality solar and wind power, by making it easy to sell energy (purely on a per-Joule basis) on the pan-European market.
Meanwhile, there is no centralized capacity market or centralized incentives for black start and grid forming functionality.
> Meanwhile, there is no centralized capacity market or centralized incentives for black start and grid forming functionality.
There absolutely is. Look up terms like "Frequency Containment Reserve" and "automatic Frequency Restoration Reserve". The European energy market takes transport capacity in account, and there is separate day-ahead trading to supply inertia and spare generating capacity. Basically, power plants are being paid to standby, just in case another plant or a transmission line unexpectedly goes offline.
Similarly, grid operators offer contracts for local black start capacity. The technical requirements are fixed, and any party capable of meeting those can bid on it.
It's quite a lucrative market, actually. If during the summer a gas plant is priced out of the market by cheap solar, it can still make quite a bit of money simply by being ready to go at a moment's notice - and they'll make a huge profit if that capacity is actually needed.
No, there isn't. FCR market is not pan-European, and even where it's in place, it's basically in the name only. It's basically only countries that already use rotational generation, so it's not really a stretch for them to participate.
Spain and Portugal are not members, btw.
And the same applies to capacity markets. I believe, there is a plan to come up with a plan for it by 2027.
> Similarly, grid operators offer contracts for local black start capacity. The technical requirements are fixed, and any party capable of meeting those can bid on it.
And I don't believe there are ANY solar/wind plants that have black start capacity in Europe. The current incentives structure makes that a near certainty.
> there is no centralized capacity market or centralized incentives for black start
There certainly is in New Zealand, although the dollar amounts are quite small. If your countries regulator doesn't incentivise the capability, I believe that is a fault of your regulator.
Transpower (NZ) says:
FYI: here's a list of other contracted services that are procured to benefit network reliability/restart/resiliancy: https://www.transpower.co.nz/system-operator/information-ind...The UK has pumped storage facilities that we can use to black start the national grid.
Connect with Ireland
Isn't that backwards? When grids interconnect via DC wouldn't there no longer be a need for the grids to synchronize frequency?
With the DC interconnect, your DC to AC conversion equipment would need the capability to provide synchronized power to the generator you are trying to start. With the synchronized grid tie, your are pulling the generator into the running grid.
Does the source of "truth" matter? You still have to turn everything off, bring your generators up, each in sync and then allow load back on slowly.
The UK keeping its own time just makes things easier for it IMO.
A synchronous interconnect provides not just a source of truth but also stabilizes your grid frequency. If you have an isolated grid you have to match generation to demand to keep the grid frequency stable. If you have a 1GW interconnect that means you can mismatch generation and demand by up to a gigawatt and still be fine. I imagine that makes for a much faster startup procedure
If the isolated grids all each synced to a source of truth first (GPS?), would it be possible for them to immediately connect afterwards?
You can connect two running grids. Earlier this year the Baltic countries disconnected from the Russian grid, and synchronized and then connected to the European grid.
I imagine you can get close enough by syncing to a shared time source like GPS or the DCF77 signal, as long as you communicate how the phase is supposed to match up to the time source. Or at least you could get close enough that you can then quickly sync the islands the traditional way.
The question is if it's worth the effort and risk. Cold starting a power grid is a once in a lifetime event (at least in Europe, I imagine some grids are less stable) and Spain seems to plan to have everything back up again in 10 hours. Maybe if the entire European grid went down we would attempt something like that by having each country start up on their own, then synchronize and reconnect the European grid over the following week.
https://en.wikipedia.org/wiki/Baltic_states_synchronization_...
Nothing that complicated. You just carefully synchronize the state of the grid on both sides of the interconnect, and when they are perfectly matched, you throw the big switch.
It's a bit difficult in another way: Obviously 50/60hz is not such a high speed that it's difficult to synchronize.
The harder part is this: To pump power into the grid you lead the cycle ever so slightly, as if you were trying to push the cycle to go faster. If instead you lag the phase the grid would be pumping power into you.
That lead is very very small, and probably difficult to measure and synchronize on. I would imagine that when the two grids connect everything jumps just a little as power level equalize, it probably generates a lot of torque and some heat, I would assume it's hard on the generator.
From a physics point of view, by leading the cycle you introduce a tiny voltage difference (squared), divided by the tiny resistance of the entire grid. And that's how many watts (power) you are putting into the grid.
Yes, but that actually makes it harder to start up.
To synchronize the isolated grids, they all need to operate with an exact match of supply and demand. Any grid with an oversupply will run fast, any grid with an undersupply will run slow. When it comes to connecting, the technical source-of-truth doesn't matter: you just need to ensure that there will be a near-zero flow the moment the two are connected - which means both sides must individually be balanced.
And remember: if you are operating a tiny subgrid you have very little control over the load (even a single factory starting up can have a significant impact), and your control over the supply is extremely sluggish. Matching them up can take days, during which each individual subgrid has very little redundancy.
On the other hand, the interconnect essentially acts as a huge buffer. Compared to the small grid being connected, it essentially has infinite source and sink capacity. For practical purposes, it is operating at a fixed speed - any change is averaged out over the entire grid. This makes it way easier to connect an individual power plant (it just has to operate at near-zero load itself, move to meet a fixed frequency target [which is easy because there is no load to resist this change], and after connection take on load as desired) and to reconnect additional load (compared to the whole grid, a city being connected is a rounding error).
This is literally like Factorio.
If your Factory uses too much power, theres not enough energy to run the power plants generation, which decreases your power production. Death spiraling until theres no power.
You have to disconnect the factory, and independently power your power plants back up until you have enough energy production to connect your factory up again.
Capacitor circuit network warning for alarm for "main grid is drawing on this bank - bad things may happen if capacity is not increased."
Another "trick" is those burner inserters are black start capable. They can pick up fuel and feed themselves to keep running without an electrical network.
I also tend to put Schmitt triggers in low priority areas. They've got a battery on the main grid next to them and if the battery drops below 50% power they remain off until it goes back above 75% power.
Can't wait for a Factorio mod that simulates the grid frequency and damages equipment when it drifts too far!
https://www.reddit.com/r/factorio/comments/k16oom/better_pow... discusses the idea a bit.
That's a mod, not stock. It does two things: feeds the inserter from either end (something I think should be stock--it's a machine that grabs things, why can't it grab fuel it can reach??) and provides a trickle of power to unpowered ones (nice from a standpoint of not having to fuel them when you put them down, but it also adds blackstart capability.
As for 50%/75% triggers--the game doesn't model start/stop problems, only fancy circuit setups would give a hoot about being fed by flickering power. (But as a human....I was out adding to a big accumulator bank at twilight. Far away the bugs had a base close enough to my laser turrets that they kept attacking. The sun was powering my base but didn't have enough for the turrets. The whole bank would flicker for every bug. Usually the electrical indicator on the accumulators is a good thing.)
Burner inserters are stock. https://wiki.factorio.com/Burner_inserter
> The burner inserter is the most basic and slowest type of inserters. It is powered by burning fuel, compared to the more advanced inserters which are powered by electricity. It will add fuel to its own supply if it picks any up, which makes it useful for filling boilers with coal. This has the advantage that it will continue working even if the power fails, as opposed to electrically-powered inserters which will be unable to function.
In particular, when power demand drops below the supply everything starts running slower, which in turn means that the electrical inserters used to feed coal into the boilers run slower which drops the rate of electrical production. Burner inserters don't have that problem.
---
Schmitt triggers are really easy with stock multi combinators now. It isn't so much a "things have issue with flickering power" but rather "turn off the coal feed to the steel furnaces that would go to the electrical station instead" and "make sure that the coal unloading station doesn't brown out" and "turn off power to the electrical furnaces and labs to make sure that the coal mines don't dip in production rate when power dims".
The Schmitt trigger also makes for more reasonable "where is there excess oil that I should produce solid fuel from?" There's the optimal, but sometimes in the processing you can't crack any more crude to gas because the heavy oil is backed up, so turn on the appropriate cracking station when {conditions} and turn it off when {conditions}.
Burner inserters are stock, I didn't realize the fuel leech was now stock. It didn't used to be. And the ability to slowly grab fuel without any source of power is definitely mod. And the mod can leech fuel it's not actually handling--a burner inserter pulling from a furnace will take fuel from the furnace for it's own operation.
I'm just saying what's the need for a gap between on and off? How does the bounce harm anything in the game? Simply put the non-critical stuff behind combinators that will switch off at x% of power. Real world machinery wouldn't like that but the game factories don't care. Nor do the refineries--in your oil case, plonk down a tank of heavy oil, turn on the crackers when it's above a certain level.
The fuel leach has been that way for a long time.
https://wiki.factorio.com/index.php?title=Burner_inserter&ol... (edit from 2016)
> Since v0.10.0, any Fuel items picked up by a burner inserter will also be used to power the inserter. This makes it useful for:
> Automatically loading Gun turrets from a Transport belt, where one side of the belt is filled with magazines and the other with Coal.
> Filling Boilers with Coal. This has the advantage that they will continue working even if the power fails. This is not the case for electrically powered inserters.
https://wiki.factorio.com/Version_history/0.10.0#0.10.0
> Burner inserter will use item with fuel value for itself when it has empty inventory.
0.10.0 was released 2014.
---
The flickering isn't a problem (though a high frequency flicker can caused the power chart to be difficult to read).
A consistent drain on power that is causing capacitor levels to drop below some threshold suggests that certain things should be turned off before the problem becomes cascading when capacitor supply drops to 0.
Except that in many cases the "consistent" drain is simply night. (Or, day, on Fulgora.) Other than making the chart easier to read what's the case where a 50%/75% trigger is superior to a straight up enable if > 50%?
And apparently I misunderstood the mod, it leeches from things it could pick up but doesn't. It was probably written before it became stock and thus contained some out of date text.
In my server I hooked up a sound alarm to a set of capacitors. Too low of a charge indicates higher power consumption than production, allowing you to unplug certain low priority loads. I also have some emergency coal generators ready to go at the flick of a switch if needed.
Man I need to go play some more.
Same with Satifactory, The larger powerplants need a lot of energy for their infrastructure to run and an overload will trip breakers and shut the whole grid down, a naively designed grid death spirals very easy. My factory was needing increasingly complicated black start systems so I started putting the powerplant infrastructure on a self contained islands. something a factory overload would not trip, it was something like one coal power plant can run the machinery needed for itself and 8 grid power plants.
Thinking about this makes me wonder how all real world power plants aren't designed to survive a grid outage? Why would you ever need to black start a real power plant? Like, can't you design it so that it disengages from the grid rather than shutting down in a catastrophic outage like this? Or make it only take on a fraction of the load or something?
The problem is that it's still steam pushing spinning rust. You can't instantly scale up or down, let alone do so in an organized fashion. If you were happily generating 1000MW out of 1500MW max, there's very little you can do if a power line goes offline and you're suddenly connected to 2000MW of load - or only 250MW. At best it's going to take tens of seconds to adjust, which in practice means you're forced to dump the entire load because in the meantime your output has deviated so much that you're causing serious damage to downstream equipment - or your own. And load shedding isn't really an option because you're now operating as an island, which means you have to instantly figure out how the grid is now connected, what the current supply/demand is, which neighborhoods should be turned off to best match the pre-incident demand, and what the impact on the local grid is - there are way too many variables there to be able to respond in a fraction of a second.
Starting back up from zero is significantly easier, as you are completely isolated and have zero load. You turn the power plant on, and start slowly adding local load to ramp up. Synchronize with neighboring plants where possible to build the grid back up. The only issue is that a power plant needs a significant amount of power to operate, so you need something to provide power before you can generate power. In most cases you can just piggyback off the grid, but in an isolated black start it means you need a beefy local independent generator setup. That costs money and it's rarely needed, so only a few designated black start plants have them, paid for by the grid as a whole.
Disengage from the grid and do what with the vast amount of power that suddenly has no place to go? That kind of power tends to break things. Powerplants have big cooling towers because disposing of heat is problematic.
And you're assuming that there is a throttle setting that lets the plant produce so little power that it only runs itself. Think of the Falcon 9--the landing is hairy because it's impossible for it to produce less thrust than it weighs. The engine will go out if you try to throttle it too much.
It's hard to design a plant that can handle a drastic load reduction without shutting down entirely.
I've been thinking EXACTLY this while listening to FM radio for the last 7 hours (we didn't even have phone line)
They should allow you to build automatic rate-of-change-of-power-generation breakers...
Factorio has logic circuits which allow you to do this.
One of the Administrators of the REN, the Portuguese electric supplier is currently giving a press conference. Confirmed they are in scenario of restart from black start.
- Cause of event not known yet.
- They noticed power oscillations from the Spanish grid that tripped safety mechanisms in the Portuguese grid. At the time, due to the cheaper prices, the Portuguese grid was in a state of importing electricity from Spain.
- They are bringing up multiple power systems and the Portuguese grid is able to supply 100% of needs if required. It was not configured in such a state at the moment of event.
- They had to restart the black start more than once, since while starting, noticed instabilities in some sectors that forced them to restart the process.
- Time for full recovery unknown at this time, but it will take at least 24 hours.
It seems Spain lost 15GW of load, but is still running 10GW of load: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...
Would this suggest the grid hasn't snapped apart, or is it just not possible to tell from the data?
Coal, pumped hydro, and nuclear generation all went to 0 around the same time, but presumably that's those sources being disconnected from the grid to balance demand? https://transparency.entsoe.eu/generation/r2/actualGeneratio...
They're definitely doing a black start:
https://x.com/RedElectricaREE/status/1916818043235164267
We are beginning to recover power in the north and south of the peninsula, which is key to gradually addressing the electricity supply. This process involves the gradual energization of the transmission grid as the generating units are connected.
I see load dropping to zero on that graph, or rather, load data disappears an hour ago.
If the grid frequency goes too far out of range then power stations trip automatically, it's not an explicit decision anyone takes and it doesn't balance load, quite the opposite. A station tripping makes the problem worse as the frequency drops even further as the load gets shared between the remaining stations, which is why grids experience cascading failure. The disconnection into islands is a defense mechanism designed to stop equipment being too badly damaged and to isolate the outage.
BBC reporting the head of Spain's electricity grid saying restoring power could take "between six and ten hours": https://www.bbc.co.uk/news/live/c9wpq8xrvd9t?post=asset%3A85...
Interesting, but in terms of load I think think the data may just be delayed by ~1 hour. Switching to UTC, to avoid timezone confusion, it's currently 13:10:
Last actual load value for Spain at 12:15: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...
Last actual load value for France at 12:00: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...
It may also be a measurement artifact. Looking at the generation by type page,
https://transparency.entsoe.eu/generation/r2/actualGeneratio...
Everything dropped to zero except wind and solar, which took huge hits but not to zero. I expect those have been disconnected too, as they cannot transmit to the grid without enough thermal plant capacity being online, but if the measurement at some plants of how much they're generating doesn't take into account whether or not they were disconnected upstream they may still be reporting themselves as generating. You can't easily turn off a solar plant after all, just unplug it.
Either that, or they're measuring generation and load that's not on the grid at all.
Probably they are estimates of not grid metered generation assets based on wind speed and solar production, at least in the UK nearly all solar is 'estimated' because it is not measured directly (apart from larger sites), at least in real time.
Rooftop solar for example just shows as a reduction in demand, not 'generation' per se.
This also true for private wind power. Britain has a measurable amount of hill top farms where it just makes good economic sense to install a wind turbine and get free electricity. But we don't meter it, it shows up in charts as an absence - on a windy afternoon maybe Britain is seemingly consuming 4GW less electricity than it "should be". If the wind drops that load reappears on the grid and must be handled by existing infrastructure.
None of this gear is suited to a black start. If you had total grid loss for a month you could doubtless rewire it to power the farm when it's windy despite no grid, maybe even run some battery storage for must-have services like a few lights so they keep working on still days but you could not start the grid from here.
> Thermal plants require energy to start up
It's not just about the power. System components cannot be brought to operating temperatures, speeds and pressures faster than mechanical tolerances allow. If a thermal plant is cold & dark, it can take days to ramp it to full production.
That's true of some kinds of thermal generators, but not all. Simple cycle gas turbines can come up very quickly (think jet engines). Or your car's engine.
> Simple cycle gas turbines can come up very quickly
In some cases yes. Modern combined cycle plants can take as little as 30 minutes to ramp to full output. Older designs can take upward of 4-6 hours.
If you have steam as an indirection, that's when things take a really long time. Natural gas turbines are a more direct cycle.
Combined cycle turbines do have a steam component (that's where the word "combined" comes from). The waste heat from the combustion turbine front end is used to make steam in the back end.
Combined cycle turbines are not primarily powered by steam. It is a secondary consequence of their operation. This contrasts with nuclear and coal plants where steam is the prime mover.
The steam part of CC systems shows hours and hours aren't inherently needed to get a steam power plant in operation. For that matter, warships with steam propulsion show the same thing, I believe.
That's what they said, though
Can thermal stations be kept running without any laod?
This capability is called "Trip To House Load".
Sure. But not cheaply. And ramping up the power still takes time.
A true black start has several factors (which make it difficult and notable):
1. The grid has to fully collapse with no possibility of being rescued by interconnection
2. As a result, a generation asset has to be started without external power or a grid frequency to synch to
3. An asset capable of this is usually a small one connected to a lower voltage network that has to then backfeed the higher voltage one
4. Due to the difficulty of balancing supply/demand during the process, the frequency can fluctuate violently with a high risk of tripping the system offline again
None of this applies in yesterday's case:
The rest of the European synchronous grid is working just fine.
News reports stated Spain restored power by reconnecting to France and Morocco.
By reestablishing the HV network first, they can directly restart the largest generation asset with normal procedures.
As they bring more and more load or generation online, there's little risk of big frequency fluctuations because the wider grid can absorb that.
Just to add, I was at a University campus when the entire building's electrics went out and there was a significant pull due to relatively powerful computers in every room. They initially tried to bring the building back online altogether and failed. Then they tried to bring it back in sections and failed too. In the end they ended up going into each lab, turning every computer off at the wall to bring each lab's power back, and then turn each computer on one by one.
I can only imagine the difficulty of bringing large parts of the grid back online, that rush current must be immense.
Yup. I used to work with a factory that had a bunch of really big machines. Turn everything on at once and the transformer out on the pole self-destructed. Note that the breakers didn't pop--the startup transient was short enough. The power company wasn't happy.
Or look at Apollo 13. The astronauts had turned off everything possible because they had lost their generator and only had their batteries. And it took a lot of furious planning by the guys on the ground to come up with a sequence of turning things back on that didn't cause the peak draw to go too high. Can't go too fast or it trips. Can't start too early because the power is limited, but can't start too late because the systems have to be up when they hit the atmosphere.
I did a project where we predicted transformer failure (they blow up!) from changes in the oil that they have in them (its insulative properties suddenly change). This was 30 years ago so it's all a bit fuzzy, but the one thing that really stuck with me was the story that the SME that we were working with had about the UK grid teetering on the edge of failure during the "great storm" of 1987. His telling was that they really were unsure if they would get it back at all!
> It's one reason it's a good idea to always have some cash at home.
More than cash it was important yesterday to have the following in case it would have lasted longer:
- a battery powered am/fm radio with spare new batteries
- some candles and matches
- food reserves for a few days that don't need refrigeration: bread, anything in can, pasta, rice...
- some kind of gaz or alcohol stove, dry wood or bbq charcoal: you can always make a fire in the middle of the street where there is no risk of burning things around.
- water reserve (I always have like 24L of drinking water) and since I hate waste I regularly fill jerrycans when waiting for hot water in the shower that I use for manual washes (kitchenware or gears).
Does solar power make this process easier or harder? I know that with thermal plants you have a spinning mass that you have to synchronize, and phase shift is used to assess how hard the plant is working (and whether to trip a disconnect as we see here)
But with solar, how is the synchronization provided? In like a giant buck? Or in software somehow? Does the phase shift matter as much as in the electromechanical systems?
My intuition is that solar would make the grid harder to keep stable (smaller mass spinning in sync) but also may offer more knobs to control things (big DC source that you can toggle on/off instantly.. as long as sun is out). But I don’t actually know.
Mike_hearn's comment was grey but was correct: phase following is indeed done through software in the inverter. Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat.
Currently the main driver of battery deployments is not so much energy price time arbitrage as "fast frequency fresponse": you can get paid for providing battery stabilization to the grid.
(for the UK not Spain: https://www.axle.energy/blog/frequency )
Oh that is fascinating.
So if you have a smarter solar panel, or a smart battery, you can stabilize the grid. I’m assuming that all of the traditional software complexity things in distributed systems apply here: you want something a little bit smart, to gain efficiency benefits, but not too smart, to gain robustness benefits.
My intuition is that bringing the market into it at small timescales probably greatly increases the efficiency significantly but at the cost of robustness (California learned this “the hard way” with Enron)
> Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat
If the electronic controller is “ahead of” (leading) the grid, then that heat would come from the solar plant; if it is “behind” (following) then that heat comes from the grid. Is that right? And likely, solar plants opted for the simplest thing, which is to always follow, that way they never need to worry about managing the heat or stability or any of it.
I wonder if the simplest thing would be for large solar plants to just have a gigantic flywheel on site that could be brought up via diesel generators at night…
Most solar and wind plants follow the inertial lead of the thermal plants. They can't synchronize without enough thermal generation being online. Supposedly there are efforts to change that, I don't know enough about grid engineering to say how far along that might be in Spain.
> But with solar, how is the synchronization provided? In like a giant buck? Or in software somehow? Does the phase shift matter as much as in the electromechanical systems?
If you mean how does solar detect phase and synchronize to the grid: https://en.wikipedia.org/wiki/Phase-locked_loop
If you mean how does solar act to reinforce the grid: search for terms like "grid forming inverter vs. grid following inverter" though not all generators are the same in terms of how much resilience they add to the grid, esp. w.r.t. the inertia they do or do not add. See e.g. https://www.greentechmedia.com/squared/dispatches-from-the-g...
Harder mostly, See the frequency is set by huge rotating masses in the form of generators, and when the supply and demand is matched the frequency and voltage are stable, when demand dramatically increases it pulls the frequency and voltage down, which is effectively slowing the generators down as load / magnetic drag increases with current drawn. Having large inertial masses spinning actually helps smooth out frequency changes. whilst large solar farms can and do syncronise with the grid, they are reactive and do not add the same smoothing effect as humungous spinning masses.
Low Grid frequency & voltage can cause an increase in current & heating of transmission lines and conductors and can damage the expensive things, this is why these systems trip out automatically at low frequency or low voltage, and why load shedding is necessary
> Harder mostly, See the frequency is set by huge rotating masses in the form of generators
I'm not saying you're wrong, but this isn't obviously correct to me.
Since solar going to a grid is completely dependent upon electronic DC->AC conversion, I would expect that it could follow a lot greater frequency deviation for a lot longer than a mechanical system that will literally rip itself apart on desync.
the person you replied to is mostly wrong.
The real reason that small scale solar PV is grid following (i.e. it depends on an external voltage and frequency reference) is that this ensures power line safety during a power outage. That's it.
An inverter can be programmed to start in the absence of an external reference and it can operate at a wide range of frequencies.
About 10 years ago I had a chance to work in the Utilities vertical for a power producer. I asked the same question: how is the frequency set and the answer was that the biggest power plant sets the frequency because they can produce a lot power and any other smaller generator would not be powerful enough to change the frequency.
Sure, the smaller plant gets pulled into the larger plant's inertia.
However, DC-AC converters don't have an inherent inertia. They can follow almost any frequency and phase within reason. Certainly a DC-AC converter should be able to respond way faster than any frequency/phase changes that a mechanical system can generate.
In theory, they should be able to set themselves to be ever so slightly closer to ideal so that the amount of power they have to sink is limited but are still exerting a very slight force to bring the grid back into compliance rather than continuing to add load which propagates the collapse.
I'd say a little harder to negligible now, but potentially way easier in the future.
The main difficulty is that the software of grid-following inverters tend to make them trip out very suddenly if the grid parameters get too far out of spec (they will only follow the grid so far), but once the grid is good they basically instantly synchronise.
But all large solar farms are likely to be mandated to switch from grid following to grid forming inverters eventually which will make them beneficial for grid security because they will help provide 'virtual intertia' that looks exactly the same to the rest of the grid as spinning mass does.
As the press release you linked points out, the black start plan Spain trains on uses nuclear energy supplied from France to re-energize the power plants.
"Luckily", France is at an historically high level of production capacity at the moment and the connection between the two countries was reestablished fast.
According to RTE (French network manager), the interconnection was maxed yesterday at around 3GW of power.
Sadly, while Spain is part of CESA, it's not very well connected. I wouldn't be surprised if one the takeaway from the whole incident is that more interconnections are needed.
Ukraine can share some experience with forced black starts.
Ukraine is interesting in this context because there are so many generators. In the richer parts of Odesa I've even found it hard sometimes to tell whether or not the grid power is on as literally every single building had sufficient backup generators to keep the lights on (also, many big businesses seemed to run their generators even when the power was on, I presume as a civic-minded way to add generation capacity overall and avoid interruptions when the power flips on and off).
Generators are not in sync with the grid though. And that's the hardest part.
Here's why generators were running here despite the grid being available. A generator has a very short lifetime and in order to prolong it, some owners learned to run those in the very optimal schedule. Which sometimes requires a minimum amount of time to run in a single cycle. Thus if you started it you are committed to run for X hours.
> Generators are not in sync with the grid though. And that's the hardest part.
I know. What's interesting re: Ukraine is because there are so many generators there are more options to getting power sufficiently restored for normal life than just rushing to restore the entire grid.
> Thus if you started it you are committed to run for X hours.
I'm skeptical this is the main reason. While fast starting a diesel generator is hard on it, there are other, slower, ways to start big diesel generators with minimal impact on lifespan. The blackouts in Ukraine are almost all on a schedule, so big buildings with dedicated staff and expensive generators can and do startup their generators in advance (I've personally heard this happen on occasion - big generators spooling up multiple minutes in advance of the scheduled blackout).
Also, it makes sense to turn on generators in advance anyway: gives you a chance to diagnose any issues.
> I'm skeptical this is the main reason
Perhaps not the main reason, but heavy devices do indeed require minimum amount of time running. I think small devices as well but not as long. Honestly I don't remember what we did back then: so many things were going at the same time, longevity of generators was not very well know at the time )
And if you are a mere mortal in this world, play Factorio : Space age expansion on the planet Aquilo. To learn the precious importance of reliable multi stage power bootstrapping
Black Start achieved! Not at 100% yet. Apparently we have [at least] 2 blackstart stations in Portugal, both were used.
I expect those two (one CCNG, one hydro) are the protocol for a black start, but other hydros should have the capability should those fail.
Power never went out in a country completely. At the lowest point consumption was ~40% of normal for that time of day.
Ukraine went through many black starts in the first winter of Russian strikes against energy. I guess they built a skill of recovering it quickly enough that it started happening faster and easier every next time.
I would think that renewable infrastructure could be the fix, at least if you start installing larger battery capacity to meet renewable store and usage shifts, the grid essentially is installing the resources that can also be used to respond to & contain sudden source losses and prevent cascades.
I wonder if someone could build a realistic scenario into a game -- let's say some sort of smaller scale black start for, say, a space station. And throw in a unknown computer architecture for the in-game computers so that players need to RTFM to figure out how it works.
I tried to make something like this (not power specifically) for a game jam once: https://pfirsich.itch.io/arbitrary-complexity
I took down the servers though, so you probably can't easily try it. I don't know if I added a way to configure the lobby server. I should have! It's open source though. And there is a video about that thing on my YouTube: https://www.youtube.com/watch?v=6TPgfa7LbiI
The game is bad and nothing of what we planned on doing actually made it into the game. The video is long and boring too. But maybe someone finds this cool and is inspired by this and makes a game like this.
The first 15 minutes of the game were actually about getting the ship moving, first by reading the manuals of half a dozen different ship systems and then following some procedure outlined in those manuals (parts of which were simply incorrect), maybe having to do some things in sync with your other players and stuff like that. I think it would have been cool to add multiple reactors and start them up in sync and stuff. The different ship systems were actually Lua programs that interacted via a message bus. So kind of a unknown computer architecture?
Wow this really looks interesting.
make it so! the fun of reverse engineering systems I don't understand to fix them is half the reason I joined this industry.
Awesome idea!
It’s not just restarting it that’s difficult. There are the side effects like what happened in NY: https://en.wikipedia.org/wiki/New_York_City_blackout_of_1977
> have some cash at home
For maybe the first 24 hours at a grocery store, and then not so sure. Would your neighbors sell you supplies and food? Maybe not? And so many places now depend on cashless transactions and doubtful they have pen, paper, lockbox, and safe as a contingency plan.
Would this be relevant here? Spain is connected to the French grid, and probably also Morocco.
The entire EU runs on one synchronised grid so from that perspective a single 'province' went offline, not the grid.
It would be essentially the same thing as a grid black start, except that the first breaker to close has the European grid on its primary side, instead of a freshly started generator under your control.
The complex process of configuring the transmission network to bring grid power to each power plant in succession is the same.
The continental part of the EU runs on one synchronized grid. The Nordics (except half of Denmark and, uh, Iceland, Greenland etc) runs on a separate synchronized grid.
https://en.wikipedia.org/wiki/Continental_Europe_Synchronous...
Re: probably also Morocco: yes, there is an interconnect across the Strait of Gibraltar
https://en.wikipedia.org/wiki/Spain-Morocco_interconnection
Speaking of Gibraltar, it's interesting that its electricity grid isn't connected to Spain/EU's grid. So they are completely unaffected.
I'm confused. Would the start ever have to be truly black? Wouldn't water always be driving hydroelectric turbines, generating some electricity? Solar panels generate electricity without requiring input. I understand that synchronizing AC is not trivial, I'm only questioning the part about whether the start is truly black.
cash is all well and good but if the tills dont work most stores wont be able to serve you. corner shops, vape shops and barbers will be ok though
The tills have keys to manually open them, and you can just record transactions on pen-and-paper if needed and enter them later. I've seen plenty of businesses do exactly that during power outages. Not as fast. But totally doable.
Stores with tills and freezers etc will have power for the tills but the backing network for payments probably won't be up. That's the concern with being cashless. They can accept cashxl, but no one has any.
I was able to buy some groceries and pay with card. The tills had a battery backup and the network infrastructure that supports card payments was apparently working.
That said, lots of people hit the cafés and had to resource to cash payments. There was also lots of people buying bottled water at the shops.
So basically, you could divide people in two groups. Those that took it like an extra Sunday, and those that took it like the beginning of a war or something :-D
Most people around here seemed to be balanced. Yes, a lot of people outside at bars/cafes enjoying a spare Sunday, but also preparing quietly and slowly in case it takes longer time. All the stores ran out of batteries, radios and such, at least the ones we visited, and I only managed to make one card payment around 13:30 sometime, after that internet stopped working 100% until the night. Almost no super markets were open around here, maybe one or two who had generators it seems.
>It's one reason it's a good idea to always have some cash at home.
Most places are so dependent upon electricity that they can't even take cash during a blackout. And they don't even have the mechanical machines to take a credit card imprint anymore.
The last of my raised number credit cards went away last month. Those old machines will only get a blank rectangle from me. Sad, because I did actually use one of those about 3 years ago when a rural gas station had power but no network.
Network is not required for mag-strip or emv operations, but the terminal has to be configured accordingly (and has to have some way to send batch of transactions for settling). It's less common now, but fully supported.
If the network comes back up within 24hrs.
I actually have no real clue how and where Spain/Portugal is connected to rest of Europe but could they also restart North to south with help from the grid from France?
It's wild how much modern life depends on these insanely delicate chains of events going perfectly
What does synchronization mean in this context, and why is it needed?
The grid uses AC (not DC), running at 50 Hz (cycles per second). So the voltage is going up and down at that frequency, in a sine-wave pattern.
If you try to connect another generator to the grid, it needs to be at the same point (phase) in the sine-wave cycle, so that its power contribution is added, not subtracted.
If it's not in sync, huge currents can flow, causing damage. Sort of like connecting jumper cables backwards.
The bulk of the power grid is alternating current (AC), and the frequency of the resulting sine wave needs to be synchronized with the other parts of the grid it is connected to.
Sounds like my last Satisfactory build.
Oh damn, I thought this was a problem only in Dyson Sphere Program, lol
[dead]
Can’t upvote enough! Thank you!
This guy grids!
I'm going with: never attribute to malice what can be explained by ... an incredibly complex system that can fall over even if no-one's being stupid. I would want very strong evidence before I believe this is an attack.
There is precedent for major power outages, a huge majority of which are not malicious: https://en.wikipedia.org/wiki/List_of_major_power_outages
I remember the day when the Swiss railway power network went down for a day (in 2005) because one power line was down for maintenance and someone pressed the wrong button and produced a short circuit somewhere else. It's a bit like the incidents in planes were one engine has a problem and the crew shut down the other one by mistake.
I seem to recall a tree falling in Ohio knocked out the power to NYC.
https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
A monkey took out the whole grid in Sri Lanka this year
https://edition.cnn.com/2025/02/13/asia/sri-lanka-power-outa...
> The animal had come into contact with the transformer at the station, disrupting supply to the entire country. There were no immediate details on whether the monkey survived the incident.
I'm sure the constituent atoms of the monkey survived.
I saw a video the other day of some human running and jumping on a transformer after hopping a fence, dancing on the transformer in a distribution site.
It ended as you'd expect, a bright light, a lot of curse words from the camera operator who was probably blinded temporarily.
Electricity does not care.
> Electricity does not care
Nor does Darwin
That's a horrible story.
I can't believe and I'm horrified someone actually published such a video.
I'm sorry, and i agree.
I still lament the passing of cybersquirrel1
https://cybersquirrel1.com/
If that's all it took to turn Srilanka off.Even if that country lags behind western Europe, what would it take for a few humans who intentionally cut off power.
Trying to stay on the facts, this incident is likely accidental but some people even the very workers at energy companies could send a message for, I don't know. A pay raise?
Isn't it amazing? In movies, evil terrorists take out the grid all the time (or threaten to), to terrible consequences, and they need genius hackers to do so. In reality, it's easy peasy and still, people hardly ever do it (and nobody (ish) dies if they do). Humans are good!
https://en.wikipedia.org/wiki/Winter_of_Discontent
Yes, you can try to hold the country hostage for your salary by going on strike, but that's the sort of thing that results in very energetic union-busting.
Actually sabotaging the infrastructure would result in terrorism charges, or at the very least the JSO treatment.
> what would it take for a few humans who intentionally cut off power.
https://en.wikipedia.org/wiki/Moore_County_substation_attack
Unsolved to this day.
I mean i'm pretty sure there are laws in place everywhere to avoid a single pay raise strike doing nation wide catastrophic events. Hospitals for example are usually limited in how they can strike.
That's assuming the people who want to strike care about having a job afterward. I know the state hospitals stop loss people when there's a disaster, so I keep abreast of predictable disasters and tell my wife and friends "hey you're sick, go get a doctor's note for the rest of the week" and then they're not stuck at the hospitals for days during a disaster.
Not that type of hospital.
What kind of person fakes illness expressly so they don't have to help out during an actual disaster when their skills are needed? "Whenever I hear there's a fire, I tell my firefighter friends to take the battery out of their pager so they won't have to go fight the fire." I agree we owe these people better treatment than just mandating they work heroic hours, but this only makes things worse on those who comply with the mandate.
My neighborhood and community is more important than the hospital; the friends i mentioned specifically are also radio operators, so they have more important things to do during a disaster than be a warm body at not that type of hospital. A sanitarium.
The low-wage workers can get triple-time and take weeks off "later in the year" for covering the stop-loss, so they also benefit.
I don't call fire fighters, because they are emergency responders. Please note i said what i do, not what i think other people should do. The people i tell this to are free to do with that information what they want, i don't demand it or anything.
Consider this a contextual error on my part. It's more of an "inside baseball" snippet of conversation than anything that requires judgement.
Thanks for explaining what I didn't have the context for.
That wasn't tree falling in Ohio, that was overloaded line sagged and shorted into a tree, compounded with several other factors that contributed to the grid instability and the inability of the grid operator to realize how unstable the grid was.
"Same difference" to use an Irish turn of phrase. If a short to ground takes out power to an extensive area does it mater what touched what?
Yes, it absolutely does.
A tree falling can be addressed by vegetation management and trimming. A power line sagging because of excess heat is operator error.
These are not remotely the same.
The proximal cause was a race condition in a piece of software, so as usual it comes back to you keyboard ticklers!
> keyboard ticklers
I hate this term, and look forward to using it all the time.
I think the point is that neither of those events should take out a whole city; the design is such that there is considerable redundancy in the system.
Indeed. For instance, power grids ideally operate with N-1/N+1 redundancy, i.e., the disablement of any single power line should not cause a cascading failure.
Relying on vegetation management along millions(?) of miles of lines is the definition of thinking you're in control without even knowing what risks exist. I see this every year in Texas, they trim back trees thinking they have solved the problem - except - they are allowed to trim them back I think 6-8 feet from the lines, ignoring the fact the trees are 20-40 feet taller than the lines. Nobody seems to see the risk that they left there but it's quite obvious to me, we're not eliminating risks at all. We're minimizing them at best, and even then the labor required to trim all the branches around lines every year is impossible to keep up with from an economic feasibility standpoint.
They trim back what poses the highest risk with the budget they have.
As you have identified A wider right of way costs more.
Usually for lines above some voltage, perhaps 200kV, the cost of an outage due to a tree strike outweighs the cost of additional vegetation management so they will clear the right of way wide enough that no tree can fall and hit the power line.
Around here for 130kV the right of way is still as narrow as it can be and we annually take down the riskiest trees as this is the best for our budget, which is not unlimited.
Concur, they have huge fire breaks they put high tension lines in, where the easement is easily 100' to either side of the lines. This is in Louisiana. When there's a hurricane these lines still break, but not because of trees. During heavy storms, smaller trees hit smaller (480? Some KV?) Lines that go along highways to residences. A high tension line down means a few days without power (about 3-4), a tree on a lower voltage line is usually fixed within hours.
They go through and remove damaged trees near the easements of the highway lines, as well as branches that could break into lines.
As an aside we lived on the same section of grid as the sheriff, and our power was rock solid for a few years, then he left office and now our power is better than average (at least better than our neighbors who's power line cones from the other direction).
> During heavy storms, smaller trees hit smaller (480? Some KV?)
Residential distribution voltage varies by utility but it’s usually in the medium voltage range, 5kV to 35kV, with 13.8kV being common.
Where I live, there are no trees in the path of high-voltage (400KV) transmission lines. Everything is cleared below them and the voltage lines are about 90-120ft high so even if some trees grow, they will not touch the lines.
It seems like both are a monitoring problem and both have a maintenance fix. A tree falling is more forgivable though.
Both of these are operator error issues.
Is “same difference” more common in Ireland than in Britain? It’s certainly common in the US and seems to have originated there: https://english.stackexchange.com/questions/277142/etymology...
Definitely not an Irish turn of phrase. An unfortunate import.
The tree had malicious intent. You can't convince me otherwise.
Can you blame them? Just look at how they've been treated in the last few hundred years.
I vaguely remember an episode of some thriller series when I was too young to be watching it, perhaps Twilight Zone (or similar) where someone was hearing screaming, they went out a d saw trees being cut.
They've had a grudge against humanity since the fall. Ever since—once a year—they send a clear message, "leaf" but do the humans listen? No.
There was also a race condition in some management software that hadn’t tripped in over thirty years of usage.
Having lived through the Texas electricity fiasco in 2021, I would blame cost cutting, the reckless drive for “efficiency”, and maximizing shareholder value.
In Texas, the electric providers cut staff and maintenance to maximize shareholder value. They will not have redundant systems and redundant plants out of the goodness of their hearts. The Texas marketplace actually allowed them in the odd event of an outage to charge astronomical spot prices thinking this will incentivize them to have redundant systems. This was a foolish fantasy.
Now in Texas, discussion of how to cost share redundancy have taken place. But no one wants to pay for it. https://www.texastribune.org/2023/03/01/texas-power-market-p...
Show me the incentive and I'll show you the outcome.
If you want a no fail grid you need to incentivise a no fail grid.
> were one engine has a problem and the crew shut down the other one by mistake.
So frequent it even has its own wikipedia page: https://en.wikipedia.org/wiki/Category:Airliner_accidents_an...
“induced atmospheric vibration” on 400KV lines is the current theory
Probably didn't help that before the outage hit, Spain was running its grid with very little dispatchable spinning generation, and therefore not much inertia.
Solar PV/thermal + wind: ~78%
Nuclear: 11.5%
Co-generation: 5%
Gas-fired: ~3% (less than 1GW)
> Solar PV/thermal + wind: ~78%
This is (a) incredibly impressive to achieve and (b) definitely the point at which the battery infrastructure needs to catch up in order to reduce the risk of such incidents.
> battery infrastructure needs to catch up
This non-existent technology will surely catch up very soon, I wonder what takes them so long.
The only battery available at this scale is hydro and it doesn't do very well in Spain because of droughts.
It definitely does exist, I don't know why people are weird about this: https://www.solarpowerportal.co.uk/zenobe-200mw-bess-operati...
I was just thinking that grid scale batteries could be extremely useful in a black start situation.
> very little dispatchable spinning generation
Makes the case for favouring flywheels over batteries.
Just to clarify this a little bit… flywheels are cool for absorbing transients and providing short-term hold-over during a blip but don’t have great long-term capacity.
http://claverton-energy.com/active-power-article-flywheel-en... as a smaller scale example. $330/kW but at that price it only has 15 seconds of carry-over, basically just enough to get the diesel generator fired up.
Spinning reserve in the grid is equipment that capable of long-term generation very quickly. In the case of hydroelectric dams, they will often cut off the water supply to some of the turbines and use air pressure to push the water out of the way; the generator attached to the turning essentially turns into a motor and keeps the turbine spinning. If you need to bring it online, you open the water valve and let the air out.
Similar situation with natural gas-fired simple cycle turbines. They’re sitting there running at low output. Need more? Just add fuel. For combined cycle it might take a bit for the boiler to warm up for full output but having the first stage running full tilt will get it warmed up fast.
Yeah, but the angular momentum of turbines/flywheels gives you a few seconds to cutoff consumers before the grid collapses.
In a solar grid you probably have milliseconds instead of seconds, this could be the reason why the automation failed in this case.
That's part of the UK's plan:
https://www.proactiveinvestors.co.uk/companies/news/1057463/...
Same all over, apparently we had the biggest in the world a few years back, I’d imagine it’s been overtaken by now https://www.siemens-energy.com/global/en/home/stories/irelan...
Together, the flywheel and the synchronous condenser have an inertia of 4,000 megawatt-second
That's insane, imagine if it let go.
In electrical terms 4 GJ is 1,111 kWh. That's about 15 EV batteries' worth, or about £220 worth of retail electricity (at 20p/kWh). So it's a lot compared to the usual domestic things people are used to, but not much in grid terms.
Or if you consider the Irish grid (average consumption around 5 GW) that's enough energy to power the grid for about 0.8 seconds (obviously it's not going to have enough instantaneous power output to do that, but again for a sense of scale).
If Ireland had 10 of them, that'd be 8 grid-seconds worth of energy. Although, of course, actual disturbances aren't going to be that large. A few percent imbalance perhaps?
So if the whole grid had an instantaneous 10% imbalance, one of those units could carry it for 8 seconds.
(EDIT: changed energy numbers to fit the appropriate power grid)
flywheels don't really work synchronously, though. Or at least, if so, they're not very useful as storage. Inverters can simulate inertia just fine, in fact they can simulate a much larger inertia than the corresponding power of spinning generation, leading to greater grid stability (whether connected to batteries or solar/wind, though without batteries its a little 'brittle' if you run near 100% output, as the power needed may suddenly disappear, so this is probably not good practice for the system as a whole. And you still need some give in your inertia so the grid can actually communicate about supply and demand)
flywheels are batteries, of a different kind (and density) than lithium or lead ones.
You know what he meant. Respond to the charitable interpretation of the comment, etc.
They're usually almost more capacitators than they are batteries from an electrical standpoint. They're there to rapidly smooth out the smaller jitters.
I would _love_ to understand more about how this atmospheric state impacted the vibration of power lines. Sounds exotic.
I’m no expert but I think it has to do with this effect: https://en.wikipedia.org/wiki/Vortex-induced_vibration
See also: https://en.wikipedia.org/wiki/Aeolian_harp
Edit: might be a completely different kind of oscillation than I was thinking of. https://news.sky.com/story/spain-portugal-power-outage-lates...
So I would also like to understand how this works :)
Update: BBC news says something something hot weather, EU council says it doesn't look like a cyber attack.
As confirmed by António Costa (President of the European Council) on X: https://x.com/eucopresident/status/1916859055546544517
Hot weather? Huh? It's spring
Spring is nowadays the best time for the grid and renewables. Low consumption as heating period is coming to and end, air conditioning is not required yet and the sun can be strong already providing lost if solar power output. I can imagine with a grid and a huge growth of solar generation that there are a myriad tripping stones that can upset the balance.
For areas utilizing hydro, spring tends to bring the strongest annual flows due to snow melt.
In the UK we're having one of the hottest springs on record. Spain and Portugal may be similar.
Might as well mean that they suspect the culprit, but they're not going to do anything anyway for this specific culprit, so let's all pretend it was an accident. Why make yourself look like a fool?
They already became a laughing stock once for promising the "strongest possible response" for the Nord Stream 2 sabotage [1].
[1] https://www.reuters.com/business/energy/eu-sees-sabotage-nor...
>an incredibly complex system that can fall over even if no-one's being stupid
I only have a layman's understanding of power grids, but I thought they were incredibly hardened, with backups and contingencies in depth
Are the grids at this scale really this brittle? Would there be a death toll from this?
I also wouldn't blame malice without corroborating evidence
> but I thought they were incredibly hardened, with backups and contingencies in depth
Some are harder than others, and some have random flaws which nobody can really predict.
Spain seems in the transition to renewables, so it's possible that they have some flaws because they are still in the process, or because it's something which never happened before and is unknown territory. Also, Spain had some economic problems in the last decade, maybe someone build to cheap or was even cheating somewhere.
> Are the grids at this scale really this brittle? Would there be a death toll from this?
Hospitals should have backup-systems. Traffic should be able to stop in time. I guess the most problematic parts are people stuck in elevators and other spaces which only open electrical, as also the loss of cellular phone-connections for calling helpers.
> the loss of cellular phone-connections for calling helpers.
All the mobile phone installations that I saw had power for at least 24-72hrs depending on how far from civilization they were. The carriers have backups and everything.
The problem in these kind of situations is the saturation of the mobile network, not its availability.
Hardening focuses first on not damaging equipment and second on providing energy. If things go wrong quickly enough you don't have time to react, because after a power plant disconnects you get sudden bumps in load that can trigger a chain failure near the original point. The last time it happened in Europe was in 2003, which isn't too shoddy.
What Spain's PM is saying (and is being reported by Spanish newspapers), 15 gigawatts of energy production went down all in 5 seconds. Hardening to tolerate that much of a change, that fast is a more extreme event than a grid the size of Spain's preps for.
To be fair, a big deal of those 15GW probably went down as a result of the initial outage: equipment is designed to intentionally disconnect completely when the grid is behaving badly, rather than trying to force it into submission and potentially causing serious equipment damage.
In reality this means you might lose, say, 1GW due to a transmission line failing, have a big frequency dip as a result, and then have 14GW drop offline like dominoes because they sense a grid frequency outside of safe operating parameters; disconnect as they go into safety mode; cause the frequency dip to worsen; and pull even more plants offline with them. If you're not careful, a small outage can quickly cascade into an entire grid going offline.
Not many month ago the Italians trains was disrupted by a single nail into a cable: https://www.reuters.com/world/europe/italian-rail-services-d...
> I'm going with: never attribute to malice what can be explained by ... an incredibly complex system that can fall over even if no-one's being stupid. I would want very strong evidence before I believe this is an attack.
I think we should prepare for the worst though. It's wrong to assume it's not an attack too, and until we can conclude it's not an attack we should be prepared to deal with the possible consequences and act accordingly.
I think James Burke's classic talking about the fragility of our complex interdependent systems starting the episode from the 1965 Northeast blackout is still relevant and an interesting watch: https://www.youtube.com/watch?v=XetplHcM7aQ
Every single retrospective I've seen on major power outages is full of miscommunication and people who would be fired if they worked in fast food. As well as blaming the tech boogeyman ("Our systems were on the fritz this morning")
There’s an explanation that earth’s magnetic field is now so weak that a normal solar wind ended up breaking through a vulnerable spot.
Very poorly explained right now by Space Weather News. I am waiting for an updated explanation.
Seems to be caused by fire in France that connected Spain / France
a.k.a. Hanlon's Razor https://en.wikipedia.org/wiki/Hanlon%27s_razor
As far as we know stupidity isn't a primary cause, and local authorities are also probably competent at scale.
On the other hand this is Spain we're talking about, the expectations of people giving a fuck about anything are slightly lower than average.
Or that time in 2003 when a tree fell on a power line in Switzerland and all of Italy ended up without electricity.
We live in a society filled with systems that can have "normal accidents".
I'm going with AGI awakening in some lab today.
A few months ago I was hit by a blackout literally the second I was about to start delivering a company-wide talk on AI. Everything went out - Internet, mobile networks, street lighting, the lot.
We're a remote business so it seemed like I'd just rudely dropped off the call, but as everything was down I couldn't let people know what'd happened.
Apparently it was caused by botched maintenance work affecting 30,000 houses, but the timing was so perfect I can't help thinking it was because our AGI overlords really didn't want me to deliver that talk for some reason.
Sorry I couldn't resist: https://www.youtube.com/watch?v=jnDk8BcqoR0
But there is so much highly motivated malice right now.
You can see the crash on the ENTSO-E live data: https://transparency.entsoe.eu/generation/r2/actualGeneratio...
Three quarter of the production disconnects from the grid between 12:30 and 13:00, with only a bit of solar and onshore wind sticking around.
Spain loses around 15GW of demand at the same time: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...
I don't think we're able to tell from the data if one is the cause of the other, are we? Since if production was lost, load would have to be shedded to balance the grid, and if load was lost (e.g. due to a transmission failure), production would have to be disconnected to balance the grid.
Report on a smaller event in the UK: https://www.ofgem.gov.uk/publications/investigation-9-august... (see linked pdf)
That started from a combination of a lightning strike and generator trip, but turned into a local cascade failure as lots of distributed generation noticed that the frequency was under 49Hz and disconnected itself. I suspect the Spanish situation will be similar - inability to properly contain a frequency excursion, resulting in widespread generator trips.
(I suspect this is going to restart a whole bunch of acrimony about existing pain points like grid maintenance, renewables, domestic solar, and so on, probably with the usual suspects popping up to blame renewables)
> probably with the usual suspects popping up to blame renewables
Renewables were a factor in the blackout here in Brazil a couple of years ago: the models used by the system operator did not correspond to reality, many solar and wind power plants disconnected on grid disturbances quicker than specified. That mismatch led the system operator to allow a grid configuration where a single fault could lead to a cascade (more power was allowed through a power line than could be redistributed safely if that power line shut off for any reason), and that single fault happened when a protection mechanism misbehaved and disconnected that power line. The main fix was to model these solar and wind power plants more conservatively (pending a more detailed review of their real-life behavior and the corresponding update of the models), which allowed them to correctly limit the power going through these power lines.
If you want an excruciating level of detail, the final 614-page report is at https://www.ons.org.br/AcervoDigitalDocumentosEPublicacoes/R... (in Portuguese; the main page for that incident is at https://www.ons.org.br/Paginas/Noticias/Ocorr%c3%aancia-no-S...).
The problem is that some media have a tendency to present "operators did not properly model renewables" as "renewables to blame for blackout". The cause is poor modeling, but renewables get the blame as it reinforces a pre-existing "renewables bad" mentality.
This is why solar requires redundant data links to the operator in CA now. Mine has it. During instability they can command my system to accept larger voltage and frequency excursions than would normally be allowed to prevent a "pile-on" where an excursion causes solar to make the problem worse.
Is 50Hz really that important these days – i.e. more important than maintaining power? (Honest naive question.)
If you have a switching power supply brick for your tiny USB device, not really.
If you have a large spinning inertial mass like a factory motor or a power generation turbine, it's extremely important. Imagine a manual car transmission, but there's no slip-clutch, you need to perfectly align engine with the wheels rotating at 300mph, and the inertial mass you're up against if it's not perfectly synchronized is a freight train.
That's why generators trip offline in a blackout cascade if the frequency deviates out of spec. The alternative is your turbine turns into a pile of very expensive shiny scrap metal.
Frequency is effectively more important than voltage. Or rather it collapses first. Frequency reduction for a rotary generator means that more energy is being taken out than the rotary shaft energy is being put in, so it is effectively an early warning that voltage is about to drop.
Frequency coordination is absolutely critical, via phase coordination. A large generator must not get significantly out of phase. So frequency going out of spec triggers the generator to "trip" (disconnect).
As I understand it, it's not possible for a generator to get out of phase with the grid. Sudden changes in the grid phase instantly and catastrophically break generators, power lines, transformers, and so on, but that can only occur when you connect two grids together without synchronizing them. Which implies there are two separate grids to connect.
I don't know what specific threat is addressed by tripping generators offline when the frequency deviates by 1 Hz. Are they so mechanically fragile that is already damaging to them, or is it a precautionary measure because that kind of instability is likely to precede sudden frequency or phase jumps that are damaging?
> Are they so mechanically fragile that is already damaging to them, or is it a precautionary measure because that kind of instability is likely to precede sudden frequency or phase jumps that are damaging?
Both actually. A frequency mismatch between what the grid has and what the turbine is supplying causes significant thermal losses, so you got to trip the generator anyway rather sooner than later, but a significant frequency deviation is always a warning sign that something is Massively Broken and requires immediate attention to find the cause - too low a frequency means you need to shed load immediately, too high means you need to shed generator capacity immediately.
Yeah, I think it really helps to think about the grid as basically one giant spinning machine. Everything is connected with wires instead of driveshafts, obviously, but a low frequency means that all of those machines that are burning fuel to try to spin at the right speed can’t generate enough torque to spin the rest of the machine at the right speed. And on the other side, spinning too fast means that all of the machines are burning too much fuel and need to slow the heck down right now (which can be hard to do with, for example, nuclear).
That's the point though? The generator can't be lower or higher frequency than the grid. It can be applying its maximum power into a grid that is still short of power, which is probably a better situation than if it was not doing that. Tripping a generator that is supplying power into a grid with too much power (frequency too high) at least makes logical sense. Shutting down generators on a grid with too little power (too low frequency) makes no sense, unless the generator itself risks damage. Loads should be tripped offline when there's too little power, not generators.
One of the parent posts correctly identified there are thermal issues with operating at significantly off nominal frequencies for extended time periods. System operators will specify frequency bands and minimum time a generator has to stay online for each band. For example we have to stay online indefinitely for frequencies 58.5 to 61 Hz, and can trip after 60 seconds for 57.5 to 58.5 Hz, and can trip instantly below 57.5 Hz.
Some plants the system operator has the ability to send a signal to instantly open the generator breakers to shed generation if necessary.
>> It can be applying its maximum power into a grid that is still short of power, which is probably a better situation than if it was not doing that.
> One of the parent posts correctly identified there are thermal issues with operating at significantly off nominal frequencies for extended time periods
Yeah, that's very much the problem. Power sources that burn fuel to generate electricity generally don't like being run at 100% throttle for long periods of time. In a low frequency situation you've got potentially multiple countries of generators all running flat out trying to get the frequency back to where it's supposed to be. If they reduce throttle without having something else come online to take the place of the energy they were producing, the line frequency drops further.
Nuclear has its own issue with running full throttle: once a plant has been running at 100%, it takes time for it to throttle back down and might not be possible to immediately throttle back up. There's decay heat from the fission products and there's short-ish lived (up to 40-50 hours though) neutron poisons.
By thermal issues I meant electrical heating of windings in rotor,stator,transformer due to impedance changes with frequency.
I have only worked on one steam turbine and it had no issue running with the throttle valve wide open all day long.
Every system has a maximum continuous output capacity and I would expect the controls to limit the output so as not to damage the equipment.
I’m curious what sources you are aware of that can’t run at a maximum continuous output all day
There is no such thing as a frequency mismatch between the grid and the generator. There are frequency deviations from 60 Hz but turbine and grid deviate together
Yes. The speed of AC electrical machinery is determined by the frequency and number of poles. This means generators connected to each other in parallel MUST rotate in synchronism. If the frequency changes the generator is now producing power out of phase which can cause a fault (a short circuit) and possibly damage the machinery.
Its like a three legged race. You and your partner have to run in synchronism. If either of you slows down or speeds up, the other can trip and fall over taking both of you out.
Frequency is health of the grid. If it's an entire hz off of where it should be something is massively wrong. Most generating equipment will trip on a deviation that bad.
Absolutely. The frequency needs to be maintained between 49.8 and 50.2 Hz, or the situation starts spiraling out of control freakishly fast.
What I wanna know is if I could/should have a plug-in box that detects the frequency is drifting and the grid is unstable —-> high risk of imminent failure.
Might even give you clues about something big tripping offline.
Probably lots of false alarms, but if it an outage is particularly bad for you, good to know as soon as the system operators do.
Obviously over the internet could work too, but who wouldn’t love their own box?
> Probably lots of false alarms, but if it an outage is particularly bad for you, good to know as soon as the system operators do.
When the phase gets pulled down hard like what nearly happened in Texas and what probably happened here, it'll go from looking like the background noise of phase changes to catastrophic in just a few seconds. It isn't like you'll get warning an hour ahead of time. You'll probably notice your computer monitor going dark before your grafana graph refreshes.
no worries, the energy utility does that for you. they care, because if they don't, their hardware bursts into flames.
> What I wanna know is if I could have a plug-in box that detects the frequency is drifting
Yes, it's not that hard. There's smart meters and plugs that have frequency measurement built in.
You can even do it with an audio cable: https://halcy.de/blog/2025/02/09/measuring-power-network-fre...
Old school electric clocks used to (still?) keep time with the grid frequency. Guaranteed to always have N cycles per day. So if things are gradually failing, you might be able to see your clock keeping worse time.
That's not going to work, because the clock will mainly be showing phase drift.
Having the grid operate at 49.99Hz instead of a perfect 50.00Hz for a day will make your clock lose 17 seconds, but it's completely harmless. That's normal regulation, not a gradual failure. The grid chooses to compensate for that by running at 50.01Hz for a day, but that's solely for the benefit of those people with old-school clocks - the grid itself couldn't care less.
A failure means the frequency drops from 50.23Hz to 48.03Hz, probably within a single second. You'd notice as your clock stops ticking due to the resulting power outage.
Yes. https://storpower.co.uk/the-day-europes-power-grid-came-clos...
if a piano is out of tune, you'll hear it, perhaps even flinch.
if a high power transformer goes out of tune... it melts or blows up (or both); it'll try to shut itself down before that. getting it back on becomes a problem if other transformers do the same thing, which is apparently what happened in the whole country.
Yes, maintaining frequency is the most important thing, provided you want your rotating power generation equipment not to explode.
In reality, power generation equipment will disconnect itself if the frequency is too low/high to avoid catastrophic failure.
Portugal has an even bigger relative drop in load, from 5852MW at 11:00 hours -> 613MW at 13:00 hours - these seem like 1 hour averages.
[1] https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...
Wow that's interesting, solar was absolutely peak at the time. Maybe related to poor frequency leading from solar powerplants coupled with some other spike that drew it out of sync?
Is this a cyber issue? Guess too close to tell, but what are possibles?
Interesting - when you realize that nuclear energy is not as reliable as solar (nuclear disappears completely around 1 pm, while solar keeps a stable input share).
Looks like they're using French nuclear to help black start the grid [1]. Solar and wind systems could have caused the instability issues that led to the Spanish Nuclear requiring isolation. It will be interesting to see the final investigation, but my bet is that "induced atmospheric vibration" is a PR deflection from a badly designed and operated system [2].
[1] https://transparency.entsoe.eu/transmission-domain/physicalF... [2] I'm not necessarily blaming the engineers, but the politicians who force those engineers to put square pegs in round holes. For example, I can imagine politicians making a short term decision to skimp on energy storage while increasing renewable penetration. Surely renewable systems must be less reliable without storage given the lack of rotational inertia?
According to [1], 3/4 came from renewables during the re-startup. Most of today's inverters are black-start ready. That is, they don't require external energy to startup, unlike nuclear, which is heavy, costly, and slow. German readers may find fefe interesting here [2]. He debunks many of these myths.
[1]: https://www.spiegel.de/wissenschaft/stromversorgung-in-spani...
[2]: https://blog.fefe.de/?ts=96ee73ea
Interesting, especially the black-start ready inverters, but orthogonal to our comments on the stability of nuclear.
If the addition of renewables leads to an unstable grid, it is irrelevant if the grid is then restarted with the help of renewables.
Yes, you're right - these are two different things and we should not mix discussion arguments. Thanks for the point!
Apparently a local grid overload near France and a cascading failure down the Spanish network, but radio and newspapers don’t agree on root cause. Of course there is a lot of noise.
For instance, one reporter asked one of the government flunkies whether it could be a cyberattack and they turned his noncommittal “maybe, we don’t know” into “government says cyberattack may be ongoing”.
Be careful of idiot reporters out there.
Edit: I’m listening to another radio interview where they are outlining the plans to bring online Portuguese dams and thermal generators over the next few hours, progressively unplugging from the Spanish supply (fortunately we have enough of those, apparently).
It should take 3-4 hours to get everything balanced with only national supplies, and they will restore power from North to South.
Yes, something similar happened in the UK a while back. Full info here: https://www.ofgem.gov.uk/sites/default/files/docs/2020/01/9_...
Key points that started it were (you can see the chain of events in the doc):
2.4.1. At 16:52:33 on Friday 9 August 2019, a lightning strike caused a fault on the Eaton Socon – Wymondley 400kV line. This is not unusual and was rectified within 80 milliseconds (ms)
2.4.2. The fault affected the local distribution networks and approximately 150MW of distributed generation disconnected from the networks or ‘tripped off’ due to a safety mechanism known as vector shift protection
2.4.3. The voltage control system at the Hornsea 1 offshore wind farm did not respond to the impact of the fault on the transmission system as expected and became unstable. Hornsea 1 rapidly reduced its power generation or ‘deloaded’ from 799MW to 62MW (a reduction of 737MW).
Curious question for someone familiar with power at grid-scale -- How granular is load shedding? And how is this measured / tracked?
In my head, I'm thinking of generators/plants, connected by some number of lines, to some amount of load, where there are limited disconnection points on the lines.
So how do grid operators know what amount of load will be cut if they disconnect point A123 (and the demand behind it) vs point B456?
Is this done sort-of-blind? Or is there continual measurement? (e.g. there's XYZ MW of load behind A123 as of 2:36pm)
I can speak for the GB case. Low Frequency Demand Disconnection (LFDD) occurs automatically and in stages when the frequency drops until it stabilises. The substations or feeders that are tripped off are not currently determined by real-time metering - instead they are pre-allocated based on their typical demand. This means that the system operator does not really know how much demand will be disconnected at any given time. If it's sunny, you could easily trip off a lot of solar generation connected on the low voltage network, causing the frequency to drop further. It is far from optimal!
> The substations or feeders that are tripped off are not currently determined by real-time metering - instead they are pre-allocated based on their typical demand. This means that the system operator does not really know how much demand will be disconnected at any given time.
This is wild. From a amateur technical perspective, it would only take a cheap hall sensor inside the transformer to have a pretty good guess of how much current has been flowing to the load.
Hell, put the hall sensor onto a board with a micro controller and a LORA transmitter and stick it to the outside of the feed line. Seems like an incredibly cheap upgrade to get real-time load data from every substation.
The nice thing about frequency based regulation is it's an inherent property of the system, so as long as you're connected to the grid you've got the info you need to decide when to turn on or off.
If you're monitoring real time power consumption you then need a whole extra infrastructure to communicate this info back and forth. Of course you then have to consider how you're going to keep that extra infra online in the event of power issues.
Frequency based regulation is only telling you that something is wrong, but not what or how to fix it.
If you find yourself in the middle of a black swan event, and 15 GW have tripped offline, you have milliseconds to dump pretty much exactly 15 GW of load, otherwise more generating capacity is going to trip offline very quickly.
If you only dump 14 GW because you used historical data (which happens to be imprecise, because today's cloud cover reduced rooftop solar output), you're still going to be in trouble. A detector scheme with sensors at every substation would allow you to do just that.
The board is not the expensive part. It's the getting reliability qualified and then having staff fit it to every substation, arrange the data links, and construct the dashboard.
I also wonder what the realtime requirement is. Data from a minute ago is fine .. except in this kind of situation, when things are changing very quickly.
Doing that at scale is tricky and requires a lot of people to participate in the mechanism, whereas the law only forces producers above a given size to participate.
The estimates we get from seasonal studies are usually close enough, especially since load shedding isn't a finesse exercise.
The situations that require load shedding usually give operators only a few minutes to react, where analyzing the issue and determining a course of action takes the lion's share. Once you're there, you want the actual action to be as simple as possible, not factor in many details.
They don't really disconnect it like that in these circumstances (as in they choose what to disconnect). As far as I know generation plants will start disconnect when grid frequency is <49Hz or >51Hz (at least in the UK) automatically as it's all starting to go very wrong. This is what causes this huge cascade effect. Roughly speak less frequency means there is more demand than supply and the other way round for higher frequencies.
This has changed a lot though, as even home batteries afaik will start discharging if they start noticing the frequency dropping to provide some support on generation. But if it's dropping too fast and too quickly it won't help.
But yes they do have very granular info on all the HV sources and how much load is on them.
Load shedding is an active measure taken when the grid operator knows that it will not be able to balance supply and demand. This is what happens in South Africa, where the operator preventively disconnects parts of the network at predetermined times to ensure network balance. As this is programmed, it is possible to rotate the areas under blackout.
In this case, we are dealing with a widespread grid incident. The various grid protection mechanisms have been triggered to prevent interconnection overload. In addition, the generators are trying to correct the grid frequency to exactly 50Hz. At 49Hz, more power must be generated; at 51Hz, less power must be generated. However, if the frequency varies too much, there are also protection mechanisms to prevent the turbines from overspeeding or amplifying frequency variations.
The grid is complex, and normally this type of incident is limited to one cell of the electricity distribution grid. A blackout is a domino effect, when a minor event triggers a chain reaction that disconnects more and more elements from the grid.
Th grid operator will have to restart or reconnect the power plants one by one, restore power to stations and sub stations. All of this must be done in a specific order before power can be restored to consumers. All of this takes time, requires resources (you need men on the ground), and the slightest error can lead to further outages.
Some consumers are prioritised, such as hospitals, transport infrastructure, telecoms and water networks. Many critical pieces of equipment have UPS systems, but these are not always designed for such long outages or have not been tested for years. There are patients with home equipment who will struggle.
This is why rotating load shedding is preferable. The outages are not too long and vital infrastructure is not affected (or less so).
In my area, "buckets" of average load are prepared ahead of time in coordination between the transporter (who gives the order to shed), the distributer (who implements the shedding operation as close to consumers as possible), and representatives of the state (who define priority areas that can't be load-shed, or that should be load-shed last).
When yhe time comes, you just shed enough "buckets" to stabilize. Load shedding is not a precise task, when you're at that point, you'd rather load a few more megawatt and be safe, than play with the limits and be sorry.
> So how do grid operators know what amount of load will be cut if they disconnect point A123
Opening a line within the grid isn't used to shed load, as the grid is mainly redundant. It's used 1) to protect the line itself (either by letting it trip, or by opening it preventively), or 2) to force power to flow differently through the network, by modifying its impedance. Opening a line from the transport network is not a way to load shed.
In this totally random example [1], opening the line A-B increases impedance in the right part of the network, forcing power to re-route through the left part, and reduces the load on another line in the right path that was overloaded.
[1]: https://excalidraw.com/#json=5l8OS96Wdke6l9YEClQt8,NJ-r2PtiE...
> Or is there continual measurement? (e.g. there's XYZ MW of load behind A123 as of 2:36pm)
The network is, indeed, monitored continually, and we do have potential network equipments failure simulations every few minutes, with contingency plans also simulated to make sure that they work in that particular case. This way, when shit hits the fan, we at least have a recently tested plan to start from.
I can only imagine the "fun" in getting those synced back up to the European grid once this is all over... That alone will take weeks
I was going to say something similar. I live in Portugal and I've heard a lot of panic/fear mongering, mainly from the techies in the co-working space I was working on and expats.
(apologies for singling out these specific groups of people - my point is that it might be worth to put down news sources like xitter, and read AP/translated local Portuguese news)
"xitter" in Portugese would be pronounced as "shee-tehr", right?
That's how some have been pronouncing it in English, too.
It's a good reminder how much noise and misreporting flies around during live incidents
[dead]
I experienced it first hand in Madrid. This was much scarier than I would have imagined.
News travelled extremely slow: phone coverage was just barely enough to receive a couple text messages every 15 minutes or so. News spread on the street, I even saw a group of 20 people hunched around someone owning a hand-held radio in the streets.
Just before power was restored, things started to get worse, as the phone coverage went completely out (presumably batteries were depleted). People were in between enjoying the work-free day, and starting to worry about how tomorrow would look like if power didn't come back.
When the northeast blackout hit in 2003 in NYC I dont remember any panic. We still had house phones and they still worked in the blackout thanks to telcos being legally obligated to give a shit about reliability.
I stopped by a friends house and we then went on a walk. Some stores were open and cash was accepted. We hung out later that night and had a few beers. The sky was amazing as there was next to no light pollution. Next day was totally in the dark as well and again, no panic. More beers were enjoyed.
The choice to move to electronic everything without having to give a shit about reliability is a failure of modern government. Move fast and break society for a dollar.
> thanks to telcos being legally obligated to give a shit about reliability.
Yeah, they don't need to do that anymore. Around me, enough towers have battery backups that I can count on 2 hours of coverage when utility power goes out (if it goes out late at night or early morning, there's usually coverage until 6-7 am when people start waking up and use up the rest of the power). I don't have a real landline, but the telco DSL would drop instantly with utility power so I don't have big hopes and I wasn't willing to pay $60/month to find out.
Around when I moved, stores would pull out the credit card imprint machines, but those don't work anymore because cards are flat. Cash might work, and I've got some, but I don't think many people in my community do; people don't have cash for the snack shack we run at my kid's sports, so I doubt they have it for restaurants and stores either. And we get frequent 2-4 hour power outages, at least one, usually two or three per year; and ~ 24 hour outages every few years. The snack shack runs during summer where electricity is most reliable, but I doubt people stock up on cash in the fall and use it all up before spring/summer; they probably just don't have any.
> stores would pull out the credit card imprint machines, but those don't work anymore because cards are flat
It's the other way around: Cards are flat because a carbon imprint doesn't afford the merchant any payment guarantee by the card issuer anymore anyway. (In other words, the "floor limit" above which cards require electronic authorization is now zero.)
At that time merchants probably still checked the signature on the back of your card against your signature on a bill. These days nobody bat an eye when I use my unsigned card. I guess at that time a matching signature would give the merchant enough confidence to process the card offline.
The merchant's confidence is irrelevant if it's not backed by a guarantee of the scheme (effectively forcing the issuer) to pay even in case of fraud.
The people operating these imprinters are sales clerks and waitstaff, not graphologists or experts in detecting altered physical credit cards. The sophistication of fraudsters has also advanced, and as a result, a system that might have been good enough in a pinch 20+ years ago isn't necessarily good enough today.
That said, in my view there's no excuse to not leverage the physical chip present on effectively all credit and debit cards these days, which is technically capable of making limited autonomous spending decisions even with both the issuer and terminal offline in scenarios like this. It probably won't happen without regulatory pressure, though.
IIRC tapping credit cards on London’s underground turnstiles use offline chip authorization to avoid delays.
It uses offline authentication (i.e. checking whether the card is authentic), but not authorization (i.e. whether it’s funded, not stolen etc.)
Unfortunately, too many cards, and all mobile wallets, don’t support offline authorization for that to be viable.
I started signing my cards with an all caps "ASK FOR PHOTO ID" 30 years ago. It raised a few questions when I would travel to the US and use them there, but was never refused a transaction.
That’s an urban legend, and stores are not required to actually do so. (And as far as I know, a thief could sign the receipt in all caps, ASK FOR PHOTO ID, and it would be a valid signature :)
In fact, even verifying the signature is no longer required in at least the US.
Signature verification also only solves cardholder authentication, not card authorization (i.e. figuring out if the card is funded, still valid etc.)
I've been signing "CHECK ID :)" for several years now and it's only a few times a year that someone notices.
Meanwhile (some) politicians in my country (Poland) and EU say that we should limit cash handling or eradicate it at all
Please point at particular people and parties without wording it as "some politicians".
In Poland alone we have far right, a lot of centric parties and some more or less leftist parties.
The most popular (by popular votes) "right wing" PiS is not "right" for the most part for several years now, had a Marxist/socialist prime minister and gave so many social benefits away to grab votes that it made "left" blush. They are "right" only on the level needed to get church-goers votes.
And, with antisemitist agenta, PiS was buying Pegasus subscriptions from NSO to spy on opposition and unvafourable journalists just in hopes that they'll get something from it.
So - which "particular politicians" are you quoting?
[dead]
Who? Please provide names.
2003 was over two decades ago. Most people didn’t have mobile phones let alone smart phones. I’ve been in Lisbon today and it’s surreal being 100% cut off from friends and family back home and a big relief power is back on, we’ve become very used to and reliant on seamless instant connection. Our mindsets and how we live our lives with that instant communication is totally different to 22 years ago.
> 2003 was over two decades ago. Most people didn’t have mobile phones let alone smart phones.
Maybe it depends on the country, but my memory of 2003 is almost every non-elderly adult I knew (in my own upper-middle class milieu) already had a mobile phone. Not a smart phone as we understand the term today, but a lot of phones back then had primitive smarts that are now largely forgotten, such as WAP/WML browsers (which maybe not many people used, but I certainly remember using one), JavaME applets (vaguely remember using them too-maybe post-2003, but higher end 2003 phones definitely could run them), vendor-specific mobile app formats such as Symbian
It must be. Most people I know got their first cell phone in 1997. That year over a few months it went from almost noone has a cell phone, to almost everyone having one.
That sounds like the sensible reaction, at the time at least.
It's interesting to think about and realize how much things have changed now though, and how reliant people are on everything, and especially their tiktoks etc. working all the time.
Some of the panic is likely related to the war in Europe too, and especially the general talk about war
> Some of the panic is likely related to the war in Europe too, and especially the general talk about war
We were just two years removed from 9/11 so terror talk was the first thing that happened. We got that news from AM radio in our cars. Still no panic.
Back in the 90's I worked at Nortel and visited a modest size Captive Office in Los Angeles. It supported maybe 20k or 30k people. I was amazed by the field of lead-acid batteries, 1.5m high x 50m^2.
Like 10-15 years ago before VOIP I sold internet/landline services door to door for a summer. My biggest selling point was explaining to people (usually who had children) that the VOIP service they switched to would not work in an emergency where power wasn't available... Worked like a charm to get people to switch back.
Similar experience in a town in Madrid's metropolitan area.
Electricty went down (something kind of frequent). My UPS kept PC up, and alarm system with sim and small UPS mantained wifi up for an hour or so.
Scary moments started when people I was in a call with in Portugal texted 'Grid is out'. Later no phone signal nor data.
At first, it might seem people running towards supermarkets an overreaction on being without TikTok for a couple of hours, but you have to live how scary it is to experience this in Europe's current political status to know 60 million people (plus industry) in three countries are out of the grid.
If you see Snowden's film (this might not be the most trustworthy source) it is exactly how CIA's agent describes the feasable attack towards these countries. Again, not a valid source, but I'd love to understand if that could be feasable.
Experience from Portugal, near Lisbon: fake news and made-up stories traveled fast! My wife called me (before phones went out) saying someone heard on the radio that Portugal was on red alert, it was WW3 (world war 3) and I think she even mentioned "missiles"! Also someone said it was a cyberattack and all Europe was off. Lots of panic reactions, many people buying toilet paper, water, candles, sausages and other canned food.
All gas stations closed because they could not sell gasoline/diesel. Today there are lines on all gas stations, people filling their car tanks and bottles..
Oh, let me tell you about electric cars! Many people had to spend the night somewhere away from home because they could not charge their cars.. My sister (with her job's electric car) had to stay the night some 200km away from home, and since the ATMs (Multibanco) didn't work, she didn't have physical money to pay for food. Luckily a stranger paid for the food (yogurt and some cookies). Petrol cars, because of their range, had better luck!
Pure fear and panic..
I can only blame the authorities (Portuguese/European) for not having contingency plans for keeping people informed, and thus letting fear spread like wildfire.
I have to ask: what's with "many people buying toilet paper" when there's the smallest probability of a disaster? Why? Why? Why?
Because you're always buying toilet paper in bulk, and everyone needs it.
Toilet paper is cheap and bulky, so stores only stock what they absolutely need. If a store is supplied once a day, it'll have something like two day's worth of toilet paper on its shelves.
Some incident happens, and people start to panic. By sheer coincidence one brand of toilet paper happens to be out of stock. The shelf space is huge, so as you walk past it you immediately notice it and think "geez, wouldn't it be awkward if that were to run out". You don't know how much you've got remaining at home so you grab a pack, just to be sure. Ten people do this, and because the packs are so massive another brand has just run out - making even more people consider picking up toilet paper.
You normally buy toilet paper maybe once every month, so if only 1 in 15 people pick up toilet paper during their panic shopping, those two day's worth of stock will quickly run out!
It's because of Rule #3!
https://www.youtube.com/watch?v=Xs8ayvLzqxY
At the point where the catastrophe has already started, it's too late to be taking up cardio, but rule #3 is one of the rare ones under your control.
I've asked the same question. I think people are so pampered in western societies that the worst thing they can possibly imagine is not being able to wipe their ass with a fresh piece of toilet paper.
What, no. The toilet paper hysteria began in Hong Kong.
And the irrational urge to panic buy toilet paper became rational since enough people were irrationally buying it.
If you are old enough to have a conversation with someone who lived through the 1930s United States Great Depression, they had to resort to tearing pages out of the Sears catalog. After that, they would hoard toilet paper. And every generation since has done the same thing. But it started with the Great Depression.
During a bad storm here many people used their electric cars to power their house. There were still places to charge it and then come back to the house to power it until the house was re-connected.
[dead]
I think this is where F-droid and Briar have a (short-lived in this instance) chance to shine. Since Briar allows communication between phones without access to the Internet, and F-droid allows to direct transmit apps between phones as well.
I wonder what similar solutions exist in the iOS ecosystem.
we tried briar during the recent mass protests in Serbia and the bluetooth connection never worked, it was unable to connect to any of the contacts that were previosly added using the normal 4G connection.
Until you have to charge your phone.
My son was at the university outside of Barcelona, and I lost contact with him for 6 hours. He was traveling home, and I knew that the roads had no working semaphores, and there were dozens of incidents...
I lost some years today.
My son is fine, thanks to a random person (“the man with a rabbit”) who just decided to give a lift to my son and his friend to the edge of Barcelona.
Glad you are all ok. There have been many stories of people supporting and helping each other, very uplifting in these troubled days.
I come from a world where I faced power outage for 17 hours for weeks. Each day there would be power at random time of the day. Life is just bleak. And when I see these kind of posts, I sympathize with them. You can't do anything in that situation. Modern world itself is so complex, you are left to thoughts. If this continues for a month, most people will go insanse especially those who rely on technology to stay sane.
I hope this comes back ASAP.
no one has emergency fm radios there? I thought that was pretty much ubiquitous. that's the first I'd go to check what was going on I think, other than my phone
From my personal experience, mostly just in the car. Here in Portugal there was a rather well-timed rail strike today, so a lot of people who would've taken the train drove instead and had a way to get back and listen to the radio. From what I could tell from the radio reporting, handheld radios and batteries were rather quickly sold out.
I've got to give massive props to Antena 1 too, which is the national broadcaster's main radio station, who stopped all normal programming and did an all-day massive report on this situation to keep people informed. From what I could tell they didn't even run any ads during that period, just all-day reporting continuously repeating key information for people who'd just tuned in.
> From what I could tell from the radio reporting, handheld radios and batteries were rather quickly sold out.
Indeed, visited the local hardware store and some bigger stores like Decathlon, and they were out of all batteries, gas-powered stoves and anything else related. Seems they ran out of it just hours after the power was cut too.
> I've got to give massive props to Antena 1 too
RNE did a great job, and together with the response of the previous crisis of Covid, I feel relatively safe as a Spain resident during a crisis. People around you are so caring as well, like when I tried to figure out how to open my garage door without power, a young guy stopped while passing by to ask if I needed help. Simple stuff, but gives a larger feeling of that we'll survive together no matter what.
I did not see much fear. I was at work and it took about two hours for us to realize the outage was not just local. The cafeteria had gas burners and served everything they could to empty the refrigerators. We all at lunch and discussed whether those who lived far away (train trip) would need to sleep at work (they might have, I don't know what happened to them). I made the relatively short 75 minute walk home across the city.
The atmosphere was quasi-festive and most people were quite relaxed, enjoying an unexpected afternoon off. Younger people filled the bars which were serving everything they could. There were long lines at supermarkets and an occasional fellow toting a box of supplies, but mostly there were just huge numbers of people in the street and completely collapsed traffic flow (the police were out in force almost immediately, directing traffic). In the part of Madrid I was in about 1/4-1/3 of the population is from South America and I suspect most of them have seen this all before anyway. The only real stress I saw was from people that need a train to get home (because the trains weren't running) and a had a walk of more than 2-3 hours.
I got cell phone signal when I was near two hospitals which were fully operational.
It was interesting that almost immediately, while I was still at work, everyone said power was out in Portugal and France too. After an hour or two some were claiming problems in Germany, but this seemed already to be unfounded rumors.
Some younger people couldn't walk home because they didn't have google maps ...
What caused it? The Portuguese prime minister, Luís Montenegro, said that the issue originated in Spain. Portugal’s REN said a “rare atmospheric phenomenon” had caused a severe imbalance in temperatures that led to the widespread shutdowns.
REN said: “Due to extreme temperature variations in the interior of Spain, there were anomalous oscillations in the very high voltage lines (400 kV), a phenomenon known as ‘induced atmospheric vibration’. These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European network.”
https://www.theguardian.com/business/2025/apr/28/spain-and-p...
"Conductor gallop" is one of several proposed causes.
https://en.m.wikipedia.org/wiki/Conductor_gallop
So, what you're saying is that literal bad vibes caused it... ;)
[flagged]
They have data and probable causes, but nobody will tell until forensic studies take place, which seems a good decission. No need to spread speculations.
A cold start from zero generation took 6 to 10 h. No casualties, hospitals and crtitical infrastructure kept working all day.
Whatever the cause, this has been a great test of the emergency protocols.
What? Every single major power outage on this list https://en.wikipedia.org/wiki/List_of_major_power_outages has taken between a day to a week to diagnose. You can read about the list of detailed theories if you click through to any of them, here's an example: https://en.wikipedia.org/wiki/2020_Sri_Lankan_blackouts
Even your minor power outages take 30 minutes to an hour for your local power company to determine what failed.
its pretty normal not to know immediately when something with this many variables stumbles. it may take a bit, but they'll figure it out.
A fire in the south-west of France, which damaged a high-voltage power line has also been identified as a possible cause:
https://www.euronews.com/my-europe/2025/04/28/spain-portugal...
A power outage in 2021 was ultimately traced to a firefighting hydroplane that damaged a high-voltage transmission line in southern France.
The similarity between that event and this early-on report is striking.
[es language]: https://www.lavozdegalicia.es/noticia/espana/2021/07/24/aver...
I'm seeing some reports saying that a significant frequency oscillation happened, which triggered automatic shut-downs, which cascaded. Could an event like this have that effect?
I suppose it makes sense that it was an automatic shutdown rather than infrastructure failing on such a wide area. And then once it's shut down, a black-start is a logistical challenge as other comments have explained.
I'm also seeing some reports about it being more likely that something happened on the east side, somewhere like the Ebro valley or north across the Pyrenees. Catalonia seems to have been particularly affected, and it's on the path of important lines coming from France. High heat at noon could have caused a line to fail and short against a tree, which would be similar to the 2003 nation-wide outage in Italy.
The grid is supposed to tolerate any single failure, even under full load. Of course sometimes the first failure is a fire or equipment malfunction and the second failure is a planning failure or someone pressing the wrong button.
Cascade failures are common. The weakest link fails, load gets re-distributed evenly, the second-weakest link fails and so on.
In theory [a flawed one] you've had enough spare capacity to survive N failures and N+1 failures are statistically unlikely because p^(N+1) is close to zero.
On practice [or with a better theory] you can't multiply probabilities in a grid system because random variables aren't independent. 30% spare capacity can go to -100% in a second.
Grey failures are harder for large systems to handle. If a chunk goes hard down that's usually easy. Something like voltage oscillations that trigger cascading failures in a sequence can lead to negative feedback loops that bring it all down.
From Le Monde live feed, RTE (French electricity network manager) declared the issue unrelated to this fire.
"Le gestionnaire français souligne par ailleurs que cette panne n’est pas due à un incendie dans le sud de la France, entre Narbonne et Perpignan, contrairement à des informations qui circulent."
Surely there is more than 0 redundancy so that one power line failure would never result in this level of catastrophe.
The 2003 US Northeast blackout was caused by the failure of only a few lines that shorted into trees. These line failures created grid instability that resulted, ~5 minutes later, in most of the Northeast losing power in a cascading failure.
Redundant systems have failures in one path quite often that you never know anything about. We get headlines when the failures correlate in the same timeslot.
> one power line failure would never result in this level of catastrophe.
Called N-1 criterion. See https://en.wikipedia.org/wiki/Contingency_(electrical_grid)#...
And it depends. During https://en.wikipedia.org/wiki/2006_European_blackout N-1 criterion was supposed to be holding, in practice not even N-0 was holding and network crashed.
Few years ago nearly entire day European network was sitting on N-0 due to multiple issues in Poland, caused by a heat wave and deeper root causes. There are many power plants and power lines where any further issue would cause Europe-wide blackout.
How very California.
you are right, but the emphasis could use a tune-up. In California, home of world-leading tech.. there are sensors and information networks, extensive electrical power lines, heavy equipment and budgets, a lot of dry and dead tress, a history of fire. So you see that California in a way is a world-quality testing lab. and the way the information travels, and the way the information is applied, could also be world-quality .. or, world-theater for government imbecility..
Warming world is so hot right now..
You can track the outage with CloudFlare's traffic radar page too:
https://radar.cloudflare.com/es?dateRange=1d
https://radar.cloudflare.com/pt?dateRange=1d
Portugal nearly reached zero.
It looks like the Iberian peninsula is relatively isolated from the rest of the CESA synchronous grid, with only 2% cross-border capacity compared to local generation. [1]
There's a map at [2]
> The Spanish electricity system is currently connected to the systems of France, Portugal, Andorra and Morocco. The exchange capacity of this interconnection is around 3 GW, which represents a low level of interconnection for the peninsula. The international interconnection level is calculated by comparing the electricity exchange capacity with other countries with the generation capacity or installed power.
[1] https://www.ree.es/en/ecological-transition/electricity-inte...
[2] https://www.entsoe.eu/data/map/
Edit : I was misreading the confusing Rte site... Entsoe actually a lot more readable. France went from importing to exporting around the incident.
https://www.rte-france.com/eco2mix/les-echanges-commerciaux-...
https://transparency.entsoe.eu/transmission-domain/physicalF...
That graph doesn't seem to make a very clear distinction between historical, real-time and predicted values... I think the event happened at 12:30 local time or so.
There seems to be some kind of recurrent daily pattern where the French - Spanish interconnect switches from Spain -> France imports to France -> Spain exports at around that time, and then back again in the late afternoon.
Yes, France was importing 884MW from Spain at 12:45, 198MW at 13:00, no net flow at 13:15, and Spain has been importing up to 265MW since: https://transparency.entsoe.eu/transmission-domain/physicalF...
One guy simulated Australia's electrical grid for 3 years to show that the entire grid could run off of >98% renewable energy using the installed renewables with just a small amount of storage (e.g. 5 hours of average demand, ~10% of demand over the entire period).
To me this indicates that in many energy markets where renewables are sufficiently built out, the only factor for why we aren't using them more is the storage capacity and grid infrastructure to handle their variability -- and instead just running stable but dirty energy systems.
There are a lot of cool mechanical grid systems like gravity batteries (e.g. weights on a pulley system) [2] or compressed or liquid air storage systems [3] which provide cheaper storage at higher capacities and durations than electric batteries
1. https://reneweconomy.com.au/a-near-100-per-cent-renewable-gr... 2. https://www.youtube.com/watch?v=trA5s2iGj2A 3. https://www.youtube.com/watch?v=fjERw-Ol-_s
It makes intuitive sense to me that a small amount of storage would make a big difference. Outside of peak use there has to be a huge amount of potential effect that never gets generated because lack of demand at that time.
Some amount of storage, significant over-provision, right balance between wind and solar; smart, well interconnected grid.
Yeah, this matches what I've been thinking too - we're way closer to a high-renewables grid than people realize, but the bottleneck isn't generation anymore, it's storage and infrastructure
[dead]
4 years ago this almost happened:
https://gridradar.net/en/blog/post/underfrequency_january_20...
https://www.linkedin.com/pulse/day-europes-power-grid-almost...
https://www.acer.europa.eu/news/continental-europe-electrici...
I remember it because power went out in at least 1/3 of Romania back then.
A similar thing happened last June, with Albania, Bosnia, Montenegro, and most of the Croatian coastline losing power simultaneously.
Definitely felt surreal to first lose power to the degree that even traffic lights were no longer working, and then to hear it's also happening across the region just before mobile networks also went offline.
https://www.reuters.com/world/europe/power-blackout-hits-mon...
~90 page report: https://eepublicdownloads.blob.core.windows.net/public-cdn-c... (beware: PDF)
It shows that even when the original issues are solved the reboot takes time due to power station being spooled down due to excessive production. I hope it is all back before telecoms start draining the batteries, otherwise things get uglier
Older readers may remeber the Northeastern blackout of 2003 in the US and Canada that was caused by cascading overloading, I believe originally triggered by high loads due to hot weather and poor vegetation maintenance under lines.
I was an an adjacent area at the time and iirc we were saved by our nuclear operator releasing some insane amount of steam to bring the supply down and avoid more overloading.
https://en.m.wikipedia.org/wiki/Northeast_blackout_of_2003
I have such great memories of the blackout. I was in college at the time, so obviously indestructible.
The first night, we had 2 grills with 10ft high flames roaring but I would say in a controlled fashion. The cops quickly asked us to not do something dumb like that anymore and we agreed.
Memories of the local party stores selling 40oz's for pennies on the dollar, as the outage would last days but the booze would not. It didn't really turn into anything bad for us in SE Michigan other than a fun story I can tell my internet pals about 20 years on.
If I recall correctly, we were also somehow the first house in the neighborhood to get power restored...I remember playing a Dreamcast with a hacked NES emulator running Rampart 2 player. I never made good on it, but I always said I wanted to make a shirt that read "I blacked out in the black out of '03"...which is probably for the best.
Older readers would remember the 1965 blackout. 2003 was only 22 years ago
Good documentary on the ice storm of 1998
https://www.youtube.com/watch?v=0ccTzHBUsYQ
> In January 1998, Montreal and the region surrounding it was hit by the most disastrous ice storm ever recorded: more than four inches of ice entombed an area larger than the State of Florida, causing trees and power lines to collapse on an unprecedented scale, leaving millions in the dark without heat (some for up to four weeks). 35 people died and damages totalled more than $5 billion making it the worst natural disaster in Canadian history.
4 out of 5 power lines to Montreal went down
And a computer system failure.
Full grid failures are almost certainly never down to a single cause.
That was a mad few days. We were stuck on campus. Water ran out by the next day. And students were using water from the campus fountain.
Portugal has no electricity as we speak. Funny enough telcos and 4G/5G are fine for now, I'm guessing batteries and diesel backups kicked in and are doing their job.
There is still no electricity, at least in my neighborhood in Lisbon. Less noise, more human voices outside. Time to meet some more neighbors I guess.
Update: still no electricity, 4G/5G is barely working, city is more chaotic than usual but not that dramatic as some media say - there are huge queues to the buses and smaller shops that are still working, more people are outside.
Yeah, we just told you that via Signal - that’s how we built the networks :)
(No relation to the other infamous Signal chat :))
There should be 4-8 hours of battery backup on every site - at least.
Wow! Battery capacity has gotten cheap.
It's always fascinated me during disasters how independent telecomm can be. Kudos for all the engineering that went into it!
I.e. even when any other conceivable dependency is down, the networks keep running.
Old-school PSTN folks looked at XKCD 705 and chuckled. Late to the party, pal.
The telephone network was designed from the ground up to be completely independent of _everything_ except fuel deliveries. If grid power is up, that's convenient, but it's totally not required.
In many places, that's because telegraph and telephone lines got there before the power grid did. Lines running along railroads connected communities that had no centralized power generation. Delco-light plants at individual farms might be the only electric power for miles, aside from the communications lines themselves. Even if the only phone was at the rail depot, it still had to power itself somehow. As those communities sprouted their own telephone offices and subscriber lines branching throughout town, the office had its own batteries for primary power, and eventually generators to recharge them. (Telegraph networks largely ran from just batteries, recharged chemically rather than with generators, for years.)
Fast-forward a century and there was just never a need to depend on anything else. As long as the diesel bowser can get down the driveway, the office can run indefinitely.
Among old AT&T/Bell/WECo hands, the devotion to reliable service goes far beyond fanatical. Many offices built during the cold-war have showers in the basement and a room of shelf-stable food, though these are no longer maintained. The expectation was that whoever was in the office when the bombs dropped, would keep things running as long as they could. And when they couldn't anymore, well, there was probably nobody left to call anyway.
https://xkcd.com/705/
Depending on country there are sometimes very strict requirements - or just traditions sometimes - around building up strong survivability in face of total loss of grid power. Including diesel and turbine generators on bigger BTSes let alone exchanges. If you drop capacity per terminal (so bandwidth) you can cover a lot more range at times which helps with mobile network resiliency.
> Including diesel and turbine generators on bigger BTSes let alone exchanges.
Or if you’re AT&T, grid natural gas backup, so your CO goes down if electrical and natural gas both go out once the batteries die. Did I mention how they didn’t build in roll-up generator connection points and had to emergency install those?
https://news.ycombinator.com/item?id=25539586
Batteries lasted almost 8h but they progressively cut down services until we barely had voice.
> we just told you that via Signal
Who are you and what's Signal?
I genuinely don't understand the sentence "Yeah, we just told you that via Signal"
We know each other.
Barely.
Like ships crossing in the night.
:D
Most base station masts have lead battery backup up to 24h - 48h.
There's a map of realtime load flow here: https://gridradar.net/en/wide-area-monitoring-system (currently shows Spain and Portugal as 'offline')
The best source for data seems to be the European grid operator themselves: https://transparency.entsoe.eu/dashboard/show
Spain's demand: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...
Spain's generation: https://transparency.entsoe.eu/generation/r2/actualGeneratio...
Spain's import/export with France: https://transparency.entsoe.eu/transmission-domain/physicalF...
The filters can be used to see similar data for Portugal
Interesting that the recovery (edit: of load graph) is going at relatively steady 600 MW/hour, it will be a while if the pace continues the same way.
Almost certainly being coordinated at that rate by adding one plant at a time, then a load region, then checking stability is holding, and so on.
Are you sure "offline" means that? Romania looks offline and when I checked their CNN they were reporting live from Spain about the blackout without mentioning Romania.
Here: https://tv.garden/ro/F83BfecjsD6BjR
The map seems to be based on monitoring stations in the different locations, so yes - it's also possible that a station is offline for other reasons (maintenance, etc).
They were probably put offline as the network is rebalanced. That just means they (Romania) don’t contribute to the network.
Our government said we're fine with our local generation and are even exporting 200 MW (possibly to our eastern neighbors?).
I definitely had no problems with electricity all of today (on the eastern side). And there was nothing in the news about local outages either.
Funny enough, there were news before the Easter holidays that they're preparing for extremely reduced demand by shutting down facilities.
10 mins ago Malaga was online, now it's offline. It doesn't look promising
Might just be lag?
In any case, if I recall correctly from a Youtube video I can't find (it was either Wendover or Real Engineering), if the grid is fully down, it takes quite a lot of effort and time to bring it back online because it has to be done in small steps to avoid over/under loading/using.
It was Practical Engineering. https://www.youtube.com/watch?v=uOSnQM1Zu4w
Very good video. Very good channel.
Idk, it might be updated with some lag
Romania seems offline as well now?
now mannheim shows offline
Remember folks, there's about 1% chance of a Carrington event occurrence every year. We're simply not ready.
Last big outage here in Spain, we got a generator, wind turbine, solar and battery. All cheap Chinese stuff as it doesn't happen often, as a result, this event was barely noticeable to us. Especially the wind turbine did well the entire day and night until the power came back a few hours ago.
What batteries did you get? We have solar and wind, but storage is a real problem.
What are the non-expensive non-Chinese alternatives?
Remember folks, civilization could end any time. We're simply not ready.
Would help if fossil fuel industry would stop pumping out anti-climate change propaganda.
Fossil fuels increase grid stability?
They’re hiding Carrington events risks ?
can't be ready for everything and anything shrug
We can be significantly better prepared for most things.
We are prepared, just have the people in the suburbs breed some more and ship them over to the decimated cities. You can think of people in the cities as the rat-wheel of production. No culture is better than any other culture, therefore all culture is expendable.
no matter how well we're prepared for anything, we can still be significantly better prepared for most things...
No, not realistically. There are physical and economic constraints. There's no reason to over-prepare either. It's absurd to remain as catastrophically vulnerable as we are though.
People are not voting this up to top because they're offline. This outage is quite massive.
[flagged]
Spain has 49 million people, which is a lot, but not 80 million.
(plus 11 million of Portugal for a total of 60 million people in the Iberian Peninsula)
ok so just remove times two then XD
I'm in Valencia and it is indeed happening here. A street parade under my windows continues nonetheless.
> A street parade under my windows continues nonetheless.
I mean, what else are you gonna do without power?
Usually?
Sex. At least that's what everyone believes
https://www.snopes.com/fact-check/from-here-to-maternity/
Well, sex and looting, and maybe inventing new music genres.
The legend is that hip hop and sampling started after the blackout in NYC. Some electronic and music stores were looted and the recording equipment eventually ended up in the hands of musicians looking to make a new sound: https://www.rollingstone.com/music/music-features/new-york-c...
How are you posting this?
Telcos have tons of backup power. They’re required to have it.
Mobile internet still works.
It works, but only some of the time now. It’s been very unstable for me since the outage started. I’m in Spain.
"Cell towers have backup generators?"
All essential infrastructure has to. Heck, if you have a landline you can probably siphon off some power from the DC component.
Cell towers have backup generators?
Most of them have batteries. How long they last depends, in my country they can typically manage 4 hours.
We had a (much smaller) regional (Lübeck and Ostholstein) power outage here in Northern Germany in I think 2018 or 19. No mobile internet until everything was restored ~6 hours later, and most of the time no other cell service either.
They still need other infrastructure to get anywhere though (routers and other networking infrastructure).
Cell towers have batteries and/or generators, with fibre connections to data centres, which also generally have batteries and/or generators.
Always picture someone with a big can of diesel filling up a generator. Experience says it could be a genny running on natural gas. Had that at work before.
I think cell phone towers use surprisingly little energy, just a few kW. So even longer term operation should be possible withs bobs backyard generator running on vodka.
But I would guess the whole network equipment would draw quite a bit, especially a modern infrastructure.
Batteries at least.
I live in Madrid, and when the electricity came back around 22:00, people started cheering and celebrating in the street. You could hear how each neighbourhood was going back online.
It’s funny to think how the moment is goes off you feel nothing, but hearing that many people produce noise and express happiness makes your body notice instantly, a sensation we often describe as electric.
Grids are tricky, because the electric socket is kind of a leaky abstraction for heavy plant machinery in turbines of thermal plants, load balancing, synchronisation, and a gazillion other things I really don't understand.
Is a grid built on renewables and batteries somehow more resilient? Solid state things tend to be less fiddly, hence my question.
I remember reading at one point in the past that renewables were actually worse for the grid due to less predictable power generation or something, but that was a long time ago, certainly pre-battery storage.
I'll give a somewhat simpler answer than Filligree has. The problem with renewable energy sources is that they are typically both highly variable and not dispatchable (i.e., controllable). The former leads to supply peaks that can exceed transmission capacity or supply lows that require compensating generation elsewhere. The latter means that energy generation can easily be increased or decreased as required, which is of course very helpful for grid management. Dispatchable generation can be increased if supply requires it, or multiple dispatchable generators can be 'redispatched' to relieve congestion in a part of the power grid (by rebalancing generation). Power from renewable sources can be decreased through curtailment, but that wastes the generated energy.
The "problem" with renewables is that they are not dispatch-able, meaning you will never be 100% sure that tomorrow or a week for now there will be sun or wind in a specific region. On the other hand, a nuclear or a coal powered plant, know months in advance if they will be available for peak load or not.
Yes, if it is sunny or windy, they can be scaled in minutes, but only if conditions are met. The inverse is true for nuclear/coal - they cannot be scaled up & down in minutes.
Yeah, hence I mention batteries specifically in my question. I guess a renewable grid will need a lot of batteries. And presumably these are highly dispatchable?
Batteries are the ultimate in dispatchable. Provided they're charged.
> Is a grid built on renewables and batteries somehow more resilient?
So this is a complicated subject in itself, and a full answer won't fit inside this textbox. Some bullet points:
- Grid stability is maintained by batteries, but not literally. The "batteries" in question are typically rotating generators, i.e. turbines, wind, literally anything where you have an electrical coupling to a lot of physical inertia. That's what keeps the grid running second-to-second; while a power plant might pretend it's outputting a constant 4MW, it actually shifts noticeably from moment to moment. The kinetic energy of the generator helps balance that out.
- Going up from the sub-second range, an overload of the generator obviously would cause the shaft to slow down, dropping the frequency and causing brownouts. Brownouts are bad and can damage the grid, so typically breakers will disconnect if it falls below 49Hz; a 2% drop.
- Baseload plants can't cope with this, as they take multiple minutes to spool up. Minimum; for something like a coal power plant, where you have to shove in additional coal and wait for it to catch fire, it's going quite a few minutes. This is what defines 'baseload'.
- Peaker power plants can increase (or decrease) their mechanical power production in a matter of seconds. These days that typically means gas turbines, though hydroelectric power is even better, and nuclear power could be used for peaker plants -- but isn't; most nuclear reactor designs outside of the navy is a baseload design. France does have some load-following designs, and we need more of those.
- Wind turbines can't increase their output, flat out, but they can decrease it (by feathering, or by using brakes). This is good enough, except this would turn them into 'peaker plants' that can't help with peaks. If we had enough wind turbines to cover 100% of the load then we'd technically be fine, but economically speaking that doesn't work; they'd be at less than 10% power most of the time.
- Wind turbines have rotating shafts, but a lot of the time they produce DC power, linked through inverters, which removes that benefit and makes them act like solar panels in effect. However, this is a purely economic issue; they can trivially be upgraded to support grid stability if the pricing scheme will pay for it.
- Solar panels are worse: They have no inertia! There is no rotating shaft there to cover sub-second usage spikes. That's where complaints about 'renewables causing reduction of grid stability' come from, along with issues like domestic solar needing to backfeed power through distribution lines and transformers that aren't necessarily designed for that.
- But batteries can absolutely help. The kinetic energy of a rotating turbine isn't actually that big; it's not that expensive to pair a solar panel with a battery to build a grid-forming system that acts the same way a kinetic power plant would.
For wind turbines, solar, and batteries, the frequency stability is doable through controlling frequency from inverters.
Some wind turbines are also internally a hybrid design that can dynamically adjust the frequency difference angle both for minimal losses in production, but also to provide frequency shifting and even artificial demand (i.e. essentially using wind power as brake)
The inertia provided by synchronous generators was in essence a natural benefit to the grid that we never appreciated previously. But now with more renewables it’s now something we have to consider and design for. Running a grid on 100% non synchronous requires a bit more effort!
Running the grid requires a lot of effort, period!
It almost sounds like running a large motor/generator with a large flywheel on it, just for the purposes of frequency stabilization and flattening load spikes, may be a good idea in a large solar installation. The downside is the mass and the bulk, and the cost of copper. The upside is zero reaction time.
It is in fact a thing: https://en.m.wikipedia.org/wiki/Flywheel_storage_power_syste...
I believe (but I am 0% expert) these are mostly but not exclusively used for adding inertia to the system, rather than real energy storage.
Close, but you're looking for: https://en.wikipedia.org/wiki/Synchronous_condenser
Apparently this spans more countries? Very strange. Possibly a cyberattack or sabotage?
Growing up in Spain I've never experienced anything like this (not there at the moment, but friends have told me over WhatsApp).
Possibly some kind of rare cascade failure where it might have been on the edge for a while, and some small event happened that tripped things, similar to the american northeast blackout in 2003. High demand, plus a power station going offline meant more demand on some interconnects, which shorted on trees and were cut off, putting more load on other lines until the entire system collapsed
Continental Europe Synchronous Area [1]
The whole Europe power grid are somewhat interconnected I wont be surprise if this knock on effect start knocking out other surrounding countries.
[1] https://en.wikipedia.org/wiki/Continental_Europe_Synchronous...
Is that a risk with the small amount of interconnect capacity?
Grids are widely interconnected. Problems on one grid can and do cascade to another.
[flagged]
Ah, yes. The dubious and evil Perro Sánchez.
Woof!
[flagged]
As of:
this is breaking just minutes ago:* https://www.bbc.com/news/live/c9wpq8xrvd9t (rolling updates)
* http://archive.is/https://www.bloomberg.com/news/articles/20...
* https://www.rte.ie/news/world/2025/0428/1509881-spain-portug...
The Spanish national operator:
* https://www.ree.es/en
* https://en.wikipedia.org/wiki/Red_Eléctrica_de_España
Red Electrica - red is Spanish for network, not the color (roja)
In Portugal here. Lost power about 1 hour ago. Reading the news while batteries last.
Amazing/curious that all the infrastructure to feed news to your phone still works. I can imagine a poorly-built systems would have some infrastructure without UPSes, and in all that chain of technology, there'll be an element or more without power. Not saying Spain/Portugal would necessarily be more prone to have poorly-built infrastructure, I assume the whole world runs on half-assery.
If their networks are built like the ones I've worked on, everything has at least a little battery, but only some of it has generators to keep going after that.
So some or most cellular towers will have generators, and their fibers will backhaul through repeaters, some or most of which will have their own generators. When it gets back to the MTSO, that will definitely have large diesel turbines on site and at least 24 hours of fuel with priority refueling contracts.
I'd expect there to be a lot of outages, for instance where all the towers in a region end up backhauled through a site where the generator fails or was never installed for some reason. But there will also be a lot of places that stay up in some capacity because, more or less by happenstance, all the fuel tank permits got approved and all the equipment actually worked.
I believe it was 10 years ago or something, we lost power in Turkey. I mean the entire country for around 10 hours.
Data, cellular etc everything kept working. But at some point I guess the generators and batteries started to fail and capacity degraded.
Turns out not everybody in the world are jokers.
A full nationwide power outage affecting not one but three [0] countries.
Sounds like a major infrastructure risk given that it is possible for more than one country to experience a full loss of power.
EDIT: Andorra is also affected, so that is three.
[0] https://www.lavanguardia.com/vida/20250428/10624908/caida-ge...
The European grid is a connected synchronous grid. Usually that would add stability but it also means if a single country's grid blacks out then their neighbours have to respond to that. Portugal and Spain's grids will be intimately connected which won't have helped.
Not something that's easy to test for.
The whole European grid is essentially connected at this point, it's a simple trade of resiliency for efficiency. People just forget about the resiliency part until something goes wrong
Yeah I've been wondering why this seems so centralized. I'd think power would be more decentralized.
ditto, definitely TIL
Looking forward to the postmortem
Accidentally dropping the production database doesn’t seem that big of deal in comparison to killing the electricity in two countries.
unless it is database of russian oligarchys bank accounts in vietnam where they hidden their stolen money. XD
Unfortunately these things usually take months, being done at the speed of bureaucracy.
It's not the speed of bureaucracy, it's the speed of doing a good job
Things like Air Crash investigations don't take a year because of paperwork FFS. Investigating things takes immense time.
Who would be responsible for writing the postmortem? Are they required to?
I think it's ENTSO-E, here's their most recent report into an incident on 21 June 2024: https://www.entsoe.eu/publications/system-operations-reports...
For that incident, an expert panel was set up in July, the interim report was published in November, and the final report in Feburary 2025: so it'll take a few months.
I would expect the Spanish parliament to have some means of summoning an explanation, via its local regulator and grid authority. https://www.ree.es/en/about-us/regulatory-framework
Not required, but engineers tend to enjoy this sort of thing. Also, since it affected some 60 million people and EU-wide grid interconnects, someone will have to explain what happened.
Definitely looking forward to this postmortem.
Wikipedia has an article following this:
<https://en.wikipedia.org/wiki/2025_European_power_outage>
https://www.euronews.com/my-europe/2025/04/28/spain-portugal...
> A fire in the south-west of France, on the Alaric mountain, which damaged a high-voltage power line between Perpignan and eastern Narbonne, has also been identified as a possible cause.
Hello from Portugal. Local news is indeed reporting that power is out across all of Iberia.
To remember: in the early hours of this morning there was a huge purchase/transfer of monero that increased the price +30% (highly unusual) and a few hours later before lunch time we see Iberia disconnected from the grid.
Monero is the favorite payment coin when ordering real cyber attacks.
A quick Google seems to point into another direction: https://crypto.news/moneros-50-price-pump-likely-tied-to-330...
yes (of course) .. It's fascinating how always people love to jump to conspiration theories .. If it were a cyberattack (and so far it isn't proven at all), it would be nothing less than a state sponsored cyber attack to be able to get down the electrical grid from 2 EU countries - and obviously no "reward" such as crypto would be involved behind the scene
Please refrain to call it a conspiracy when it is a valid hypothesis with valid facts.
It is common to use Monero as reward for specific groups performing cyberattacks. Please inform yourself on the topics and reflect afterwards on what was written.
Please read carefully what you linked. It claims theft without a victim being identified. There is no victim and there is zero evidence of being stolen coins.
Related talk in Berlin 5 months ago detailing how to shutdown energy grids using only radio waves: https://media.ccc.de/v/38c3-blinkencity-radio-controlling-st...
Not really relevant, but the talk was in Hamburg
yes really relevant/related, as that talk also mentions that a sub-1 Hz variation of the nominal 50 Hz is already enough to cause blackouts - which is also the case for the current blackout: https://x.com/Nexuist/status/191687508022891747
the talk discusses further aspects of the European energy grid which are relevant for the current situation.
Precisely, the similarity is awfully coincident.
Thank you for the correction.
I remember the talk and thought about it when I saw the news.
No conspiracy or whatever, just thinking about the complex system which is also vulnerable for some parts.
> The used radio protocols Versacom and Semagyr, which carry time and control signals, are partially proprietary but completely unencrypted and unauthenticated, leaving the door open for abuse.
You don't think the omission of encryption and lack of authentication imply a criminal absence of due diligence.
I guess some huge contracts are in the offing fix those necessary features which should have been built-n in the first instance.
How do we know there was a huge purchase? Monero can't be tracked.
Because to buy that much monero so fast is only possible with bitcoin, which is traceable by governments and interested parties.
An estimated 305 million USD of monero were purchased this morning before the grid shutdown. That was a +50% increase on the monero price, something that never happened before in the last 11 years of the coin.
Quite a record. Quite a coincidence.
How do we know there was a huge purchase? Monero can't be tracked.
... You can see trading volume on the markets though, if it was bought with something like BTC
https://coinmarketcap.com/currencies/monero/
Thank you!
How do you think this has to do with Spain though?
I am currently in my work office in Madrid, main building has electricity so I guess they have some backup generators, the kitchen however is out of service.
According to local newspapers metro network, airport and traffic lights are all down
most office buildings have gas(methane) generator backup.
Dunno about Spain/portugal, but in North America they’re usually just for elevators/alarms, partial hallway lighting & maybe water pumps (for firefighting).
Maybe a tech company with servers will pay extra for full backup, but it’s not typical.
Without fail, during every grid outage, some will fail to start and there will be elevator rescue calls throughout the city.
Some parts of Spain are slowly coming back online https://x.com/RedElectricaREE/status/1916818043235164267
The crazy thing is how is Twitter still being used to communicate these events.
Agreed.
It's crazy how momentum can carry a business.
To use a potentially controversial example, Microsoft products (Office, Windows) are still extremely entrenched despite the overwhelming majority of knowledgable people agreeing that they're on a steep downward trajectory and the alternatives have long since surpassed them.. leading to this[0] recent video from Pewdiepie...
[0]: https://www.youtube.com/watch?v=pVI_smLgTY0
Wait til you hear that yahoo and aol still exist
Here's a "real time" map of electricity production and flow between countries: https://app.electricitymaps.com/map/72h/hourly
Would be interesting to see if it will register here.
It should take 72 hours to update
I think that must be the scale so you can browse the historical data. The resolution on the slider is 1H when on 72H scale.
[flagged]
Currently says due to power outage data might be inaccurate for Spain and Portugal.
NO. data are either accurate all the time or it is scam / bad marketing.... what are you saying ?
I don't know why you are so enthusiastic about this site. Whatever, my assumption was that they are displaying the data in 1 hour intervals as the picker suggests and showing a warning currently about the affected regions is consistent with that assumption.
I saw it on twitter and had to go to BBC to make sure this isn't some joke or hoax. 2025 and we have National level power cuts?
I know it may be rare but I think some day we really need to move or mandate every single flat / home / apartment / living places to have a 12 - 24 hours backup battery included. Something that has 10K+ Cycles, durable and non-flammable. Not only does it make sure our modern lives without sudden interruption, it also solves the renewable energy problem.
Most of the time, the grid is reliable. How much investment do we actually want to prevent a few hours of downtime once in a blue moon?
And even if we do want to invest in large amount of grid storage (which we would need to anyway, if we want to transition to renewables), I'm not sure pushing this down to the individual house is sensible. Its a great way to limit economies of scale and make maintenance/inspection harder.
Mandating battery storage is nonsense. Making the regulatory system better for it is not, though; it would be good to make it possible for consumer battery systems to do "islanding" properly so they can be true backups.
Or at least not making it illegal. Regulate it all you want, make sure it's safe, but please let's let homeowners install a transfer switch if they're willing to pay for it.
In a longer outage it'd be a massive benefit to have even just a few houses here and there that retain power.
Where is it illegal?
In many countries which want to keep the system centralized.
> I'm not sure pushing this down to the individual house is sensible.
why not? Having distributed buffers ought to make the system more resilient in most cases wont it? Not to mention that these home level batteries can be used to smooth out power usage and lower peak loads.
In fact, having an EV car act as this same battery would be an even more efficient use of resources.
Why not? Maybe because I'd rather not have to spend that much of my own capital on something I'd generally never use. I'd rather not spend the maintenance on that thing I'd generally never use. And finally, I'd rather not take up the space in my home with a large thing I'd generally never use.
I'd prefer for the power coming to my house to just not go out. The grid operator can install batteries in places other than my home. The grid operator can maintain them for me. The grid operator can get cheaper loans than I can for installing them, they're staffed with supposed experts in this stuff, just have them handle it on their premises.
Batteries can be useful for more than just getting through power outages. More and more utilities are offering time of use based pricing. For people on such plans a battery can be useful even if there are never any power outages by letting you store power from off peak times to use during peak times.
Unless it's a really massive swing it'll be a gamble if it will actually break even. A useful battery install for a home is at least several thousand dollars. I'll install $10k worth of batteries and manage to arbitrage $30/mo off my electrical plan, excellent! I'll break even in 28 years, wonderful! Will the batteries still have a meaningful charge after 28 years of use? Who knows! Meanwhile I'll miss out on the opportunity cost of that $10k over that same period.
And once again, another point for growing the wealth gap. Poor people who can't afford the thousands of dollars for installing the batteries get shafted by such things instead of us assuming the grid operators will just smooth these prices out for us.
Once again in the end I'd just prefer if there were actors on the grid running massive batteries able to arbitrage the extreme spot prices and sell me electricity at a reasonable rate all the time instead of me having to actively min/max every dang thing every moment in my life.
Doing some math on that $10k for an example of lost opportunity costs, today you can get a 10yr CD at 5.5% APY. Over a decade that's a hair over $17k, practically guaranteed return. So, to properly break even you're going to need to not just save $10k worth of electricity, you'll need to save something more like $17k. You'll need to arbitrage at least $141.67/mo on average for that entire 10-year period to break even compared to what you could have had just investing that into pretty safe savings.
Personally, I'd rather toss that $10k into my kids' education funds and sign up for a fixed rate electricity plan. Hopefully that'll be better ROI.
Grid-scale batteries have pretty different economics than some batteries in my garage. Their real-estate cost is probably significantly lower than the cost for halfway decent residential. They're buying a much larger order of batteries, so their per-kWh cost is probably a good bit lower. Also, their installation cost per-kWh is also a good bit cheaper. They're probably completely fine buying/selling purely on open spot markets, meanwhile in the end I'm going to need to run my pool pump and I'm going to want to do dishes and laundry and charge my car and all the other things on some reasonable schedule so I'd like some amount of price protection on whatever time-of-use plan I get (a day of $9/kWh electricity prices will surely wipe out whatever gains you might have made). They're probably able to get cheaper loans than me, so it's easier for them to arrange the big capital investment instead of high interest loans or having to invest existing capital.
I'm not arguing the economics of batteries don't make sense in today's power markets. Far from it, I've been toying with getting into it in North Texas. But buying a few kWh of batteries and putting them in my garage probably isn't going to break even and almost certainly isn't going to make me money.
Further thinking about what you'll actually get, it looks like a decent average price per kWh of installed home battery backup is about $1,000/kWh. So if we got $10k of batteries installed, you're looking at 10kWh of energy you can store.
Let's say you're in California, which has a pretty decent swing in energy prices available to the home consumer. You'll see prices swing between $0.27 to $0.65/kWh. You've got a 10kWh battery pack, so you can arbitrage something like $0.38/kWh * 10 kWh = $3.8/day, assuming you get a full charge cycle and ignoring efficiency for simplicity's sake. $114/mo. So assuming you didn't have any loans and you're ignoring opportunity costs on that $10k, your break-even is in ~88 months or ~7.3 years.
After a decade of ~$114/mo, you'll have offset $13,680, assuming you always used a full charge and you experienced no other maintenance costs and bought it cash on hand and that energy prices didn't change and had a perfectly efficient inverter to charge and discharge the batteries and the batteries still had 100% charge capacity the whole time. The 10-year CD, which you could just forget about for a decade, stands at $17,310.
And this is for one of the few markets in the US that really offers that big of a time of use plan other than "free nights and weekends***" (with pages and pages of fine print and other fees) For instance, I just looked at a time of use plan available to me here in North Texas. 9AM-4PM I'm billed at $0.068 for the generation, $0.050029/kWh plus a flat $4.23/mo for delivery. Outside of that window I only pay the delivery cost. But that means I'm only really able to arbitrage $0.68/day with a 10kWh setup. That's 490 months or a hair over forty years to break even.
The more batteries available for that arbitrage, the less effective it is. As a grid user, I'd rather the grid own and operate those batteries. Certainly some of my outages are due to line issues between me and the substation, but I think more of them are due to line issues between the substation and its upstream. Having storage at the substation could potentially help all the customers and centralize the maintenance and safety burdens. They could do some energy cost arbitrage (although I live in the PNW and we generally don't have time of use cost distinctions, because hydro doesn't care)
I do have a 35 kWh propane fired generator onsite though, which does provide for outages regardless of where the break in the line is.
Do you have any reasons besides that?
Sure. Having large batteries is an additional fire risk in my home for once again something I'd generally never need. If these batteries are in my home but managed by the grid operator I'd probably have to allow their people into my home to service them which I'd rather not have to do if it's something I'd generally never use.
Also, this further just makes having stable electricity yet another thing in the wealth gap. Only those wealthy enough to afford the high upfront capital costs, the ongoing maintenance cost, and the space to store it get reliable electricity, fuck everyone else! Or we can just focus on investing in a stable and clean grid and share that cost with everyone, all you need is to just be connected.
But hey if I get a massive battery bank I'll have power for when the end times come. I won't be able to go get groceries anymore and eventually the fiber line and radios around me will go quiet but at least I'll be able to play videogames. For a few hours at least.
Don't get me wrong, I'm an Eagle Scout, be prepared and all that. I've got a big pile of charcoal, a chunk of propane, a camping stove, several day's supply of water and canned/non-perishable foods, some batteries for lanterns, etc. The cars all get topped off when big storms seem possible. If the big outage comes this will be more worthwhile than being able to turn on the TV for a few hours, and cost significantly less than several dozen kWh of batteries. And if the power is out for more than a week or two I'll have far bigger concerns than being able to post on Hacker News.
You're not wrong about the fire risk, but just to note, house batteries nowadays are almost always LiFePO4 -- which don't catch fire the same way LiPo batteries like to do.
They can still do so for other reasons, like a short circuit in the wiring.
Fully understood and agreed. That's why I bothered saving that for the second comment, because it's not necessarily that high of a fire risk but it is still a little bit of one. Having that much energy stored in that small of a package will always have some kind of risk of "what happens when that stored energy gets released in an uncontrolled and rapid manner?"
And that said, I do have lots of other somewhat beefy batteries around the house. They do a lot of useful things for me such as power my tools including my lawn mower, string trimmer, hedge trimmer, saws, etc. There is a massive one in the car parked in the garage. In these cases it is a useful trade off of that slight risk as I'm actually getting something normally useful out of it.
The peak load of a house is substantially higher than the per-house peak load of a city. People don't all turn on their ovens/dryers/hairdriers at the same time.
You could distribute this capacity at each house and feed it back to the grid during peak times. But is TCO of 1000 * 100kWh same price as 100MWh worth of capacity?
If you're going to have the battery anyway(a car) its hard to compete with, but once you need more capacity, I'm not sure it makes sense to distribute it quite as much.
With places that have time-of-day rates, I wonder what will happen when enough systems get advanced enough to precisely kick-off/on (or slow down/speed up) in unison at precise times.
I’ve set this up on my “smart” thermostat to speed up just before rates go up and kick off for a while once they do (and it was a pain to setup, somehow this functionality was not baked in)
In the UK, the grid used to have to manage the load when millions of people would all put the kettle on during popular TV advertisement breaks.
because installing and monitoring 1 million small batteries is vastly more work than 100 huge batteries.
How many people will turn off AC and will not bake that cake, or wash clothes, when they know they're going off their own battery vs some electricity distributor's battery shared by others?
And if someone is dumb enough to do high load stuff on their own battery during a blackout, then it's less of a problem for others. Also individual failures will cover less homes.
Incentives and consequences will be different and differently spread.
And on individual level, you can also chose whether you want this or are fine with outage. (I'm against mandating this)
Not if 1 million people install and monitor them instead of 100 people.
100 huge batteries are going to be monitored and maintained far better than 1 million.
Given how the average consumer monitors, maintains, and secures their home PC ... this is a level of assurance we want to put on the power grid?
Sounds like an incredibly expensive solution to a problem that could be tackled more efficiently at a larger scale.
I’ve read discussions about preparing for a strong solar flare basically simply involves having on hand spare equipment stored unconnected and that then would also immediately improve repairs and maintenance for regular issues. I don’t know the details but I believe the US is looking at providing funding to the utilities companies to acquire the duplicate extra equipment.
Rich countries should have a strategic transformer stockpile since they have a long lead time and should last forever in storage.
Didn't we used to pump steam around for heating and now homes have individual furnaces? If the cost for this were about the cost of a furnace I don't see why this wouldn't be viable.
I bought a generator for just this situation.
Plenty of places still do, and in those places district heating still is usually the cheapest source of heat.
That's a great example of why this is a bad idea. District heating is a lot more efficient than using individual heat sources. Especially so when the heat being used is heat that would otherwise go to waste (e.g. waste heat from industrial processes).
I get the efficiency obsession, this being an engineer forum, but aren't we talking about resilience?
I think we can have both? I don't think anyone is advocating for one giant battery. In my ideal imaginary world, there are medium sized battery parks near every transformer station. In that same fantasy, the foundations of wind turbines are also batteries, so I shouldn't pretend that this is hard science.
First-gen district heating was steam, then moved to hot water. It didn't go away [1] but there's regional differences. In cold climates there are oil based simpler backup plants that can cover for the normal CHP[2] and elctric+heat pump based generation. It's a good way to buffer and use cheap energy in periods of cheap electricty from eg high wind production[3].
[1] https://www.wedistrict.eu/interactive-map-share-of-district-...
[2] https://en.wikipedia.org/wiki/Cogeneration
[3] https://en.wikipedia.org/wiki/Thermal_energy_storage
In Polish cities there are attempts to move in exact opposite way (close individual furnaces) due to rampant pollution during winter (1000% of PM 2.5 norm level of pollution happening occasionally, over 400% PM2.5 norm lasting for weeks etc)
There is a very, very appreciable difference between moving hot steam around and moving electricity around.
The thing you said doesn't really make sense to me; I'm not sure it's an apt analogy.
There are plenty of places that still have district heating and cooling.
Mandate a battery backup. Instead of requiring the power suppliers to build and maintain reliable energy.
That is just shifting someone else's mistake into being my responsibility.
People seem to love those expensive solutions though.
I know so many people who invested in overly expensive battery storage systems for their solar. Power in Germany is expensive, but even with that expensive power many of those battery systems will never hit a positive ROI. But they’re still happy for the feeling of being „independent“.
I’ll turn 26 in a few months. The first time I experienced a power outage in my life was two weeks ago when a construction worker in our basement drilled into the wrong wall…
It's likely that battery backup solutions won't be expensive for long. Battery prices have been falling exponentially similar to Solar, and there is no reason to think it will slow down anytime soon (there is still room for improvement just from economies of scale).
I predict that home batteries will become a "no brainer" from a financial perspective (for anyone who has the upfront capital to purchase them) within the next 10 years.
I haven't seen battery price quotes for retail customers for a while now, everybody I know who wants one has one. But not too long ago a battery bank including installation was easily in the range of 800 to 1000 EUR/kWh of storage.
Battery storage may become a bit of a no brainer from a purchasing price point of view, but I don't think it will actually be more beneficial for people, especially if BEV adaption continues at a similar rate and bi-directional charing becomes widespread
You can put together a simple and fully independent solar island with 5kw/h capacity and 7kw nameplate generation for $4100. This is a plug and play system, with zero wiring and no knowledge needed. For the idea of the kind of profit margin that includes for the company, their "smart isolation switch" is $1k. Jackery has several competitors, and all of their build quality is pretty good, and the warranties are acceptable.
20kw/hr of just storage is about $10k. Every one of these companies also has standard DC input ports for bring your own solar panels, because their panels are overpriced.
The 8-Bit guy has been using such a system to maintain his studio. It is reliable, uses cheap solar panels. This stuff is commodity.
Nitpick: it's kWh not kW/h since Watt is already a unit of energy divided by time (1W is 1J per second).
Not all of us live in (semi-)detached homes, and I’m also not sure how one would implement this at the building-of-appartments level, as in, who’s going to pay for it and how do you handle the battery storage capacity for hundreds of apartments from the same building in a safe and cheap-enough manner?
I was living in Brooklyn, NY during the 2003 blackout that knocked out much of the northeastern US for a couple of days. In those days we weren't quite so dependent on cell phones and such things. People walked home, since the trains were stopped. Guys grabbed flashlights and directed traffic. We listened to the news on the car radio. After that I went to J&R and bought a wind-up emergency multi-band radio. In the intervening 22 years, I've never used it. Though at the rate things are going, maybe that will change.
“How to waste money and valuable resources” in a nutshell.
Like it or not but we are a species of social animals, you cannot live without relying on others. That's just delusional.
There are degrees of dependence on others, you make it sound as if it's a binary choice.
I don't, the comment I'm replying to does.
This is very rare. Not actually worth to make that much fuss unless you are a hospital.
Something that stores enough energy to bake cookies will always somehow be flammable.
Gravity storage of water. Not flammable, but could definitely bake cookies.
The things that convert that energy as part of the full water gravity battery unit will have flammable bits :)
Spot a European at a hundred paces by their reflex that anything happens the _government_ must do something.
If electricity is that important to you, buy your _own_ resilience. Tesla powerwalls and non-tesla equivalents have been available for ages.
Spot an American at a hundred paces proposing that you either have the money to buy batteries yourself, or else you should just eat it and suffer
A Tesla powerwall won't help with trains and traffic lights. Those are important to me as well.
unlike americans, people in europe don't just huddle in their own fortresses afraid to even take a step outside without riding in tanks.
power cuts affect more than just your home.
"Don't expect the government to do anything, give money to big corporations instead"
And then he chooses the one big corporation which has a guy in the government.
Practical Engineering YouTube channel black start video: https://youtu.be/uOSnQM1Zu4w
- "[Portugese electric network Redes Energéticas Nacionais (REN)] said: “Due to extreme temperature variations in the interior of Spain, there were anomalous oscillations in the very high voltage lines (400 kV), a phenomenon known as ‘induced atmospheric vibration’. These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European network.”"
- "The risks posed to electrical systems by big variations in atmospheric temperatures are well known in the industry, even if it is rare for problems to manifest on this scale."
- "“Due to the variation of the temperature, the parameters of the conductor change slightly,” said Taco Engelaar, managing director at Neara, a software provider to energy utilities. “It creates an imbalance in the frequency.”"
https://www.theguardian.com/business/2025/apr/28/spain-and-p... ("Spain and Portugal power outage: what caused it, and was there a cyber-attack?")
Parts of France are also affected according to Sky:
https://news.sky.com/story/large-parts-of-spain-and-portugal...
> Parts of France also appear to be affected, according to Spanish media reports, which said Seville, Barcelona and Valencia were hit by the outage.
The wording in the article makes it look like Seville, Barcelona and Valencia are in France.
[flagged]
[flagged]
The southwest of France seems ok. At least I still I have power.
France and Italy. Aliens? Russia? EMP?
There's a connected grid across most of continental europe. If Portugal's grid blacked out unexpectedly then Spain's grid could have been made unstable too, which then continue onwards to South of France and Italy.
Just a typical cascade failure because it means everything's now running with lower tolerances.
So, how do you stop this from cascading further? Did France have to cut them off the grid somehow?
A cascade failure (of any system) would propagate until it runs into a part of the system that's resilient enough to not fall over.
Several people have said the amount of power transferred between France and Spain is very low compared to the amount of power consumed inside either France or Spain, which seems like as good as reason as any that it didn't affect France. The grid in France was presumably able to absorb the sudden change in power flow without itself breaking.
There aren't as many interconnects between Spain and France proportional to generation as there are between Spain and Portugal.
Also France has a massive Nuclear base load and is usually an electricity exporter so their grid will be a bit more resilient (the spinning mass of massive turbines in Nuclear plants provides inertia to the grid itself) than the Spanish/Portuguese grid that would have had a decent renewable mix.
At the same time, electricity grids will have various measures to prevent instability spreading. Whether that's load shedding (dropping parts of the grid), removing production, or what France likely did, dropping the interconnects. This is usually fully automated.
If there were more connections to France it _maybe_ could have spread further. Nuclear power plants can be twitchy if things go south so if it had spread enough to knock 2-3 french plants offline then that could potentially have toppled France which then as a big exporter could have swept across much of the rest of mainland Europe (and maybe even the UK which might struggle to adapt to the loss from France through its HVDC interconnects if all three were maxed out)
That said, this is all very unlikely.
Neither. Apparently a local grid overload and a cascading failure, but radio and newspapers don’t agree on root cause.
Nobody fed the electricity beetles and they are on strike now.
EMP would kill electronics no?
no, most electronics is shielded by default/by law. check UL/CE requirements. but it can make circuit breakers "turn off". EMP meaning nuclear high above, there exist emp devices of size of small family van, but those have range few 100 m/ft
Madrid Mayor said
> If emergency calls go unanswered, go to the police and the fire stations in person
That's not a statement I expect to see in relation to a developed city
She's generally imitating Trump's playbook, no surprise.
A line that disturbed me with just how widespread the outage is. (And also people will die from it in all sorts of weird ways)
"In an update, Spanish power grid operator Red Electrica says it's beginning to recover power in the NORTH and SOUTH of the country."
On the bright side. People die from electricity and from operating electricity powered machines, too. So those will be spared. :)
I've seen the assertion made that we can statistically measure how many people died due to (for example) a heat wave, but we can't say for sure which ones.
I'd imagine something similar applies here. You'd have some number of deaths specifically attributable to lose of power, plus countless other deaths caused or prevented in non-obvious ways. This might be visible at a high level as a statistical outlier in the total number of deaths during the time period of an outage.
Well obviously. Computers use electricity, so if the grid goes out so does internet traffic. Local data centers might have backup power but local homes and everything in between the homes and the data center probably won't.
Seems to have briefly expanded into France and even Belgium had issues.
https://www.euronews.com/my-europe/2025/04/28/spain-portugal...
Edit: BBC reports it now also https://www.bbc.com/news/live/c9wpq8xrvd9t
Is there a La Liga game on perhaps :)
There's the Madrid Open tennis tournament affected. No Swiatek - Shnaider game until they'll get online. The restaurant is lit with candles, but the DJ has power. So there's music. He was clever.
Context https://news.ycombinator.com/item?id=43157259 / https://vercel.com/blog/update-on-spain-and-laliga-blocks-of...
OP I think is joking La Liga got the power turned off to protect their revenues.
I’m in Spain. Internet here is very unstable now also.
How are the routes looking?
On a slight side note if anyone has any good references to read on grid stability and resilience please reply.
Ex Electronic Engineer interest.
Practical Engineering on youtube has some good videos on the electrical grid. This one in particular is a good explanation on what they had to go through to get power back on: https://www.youtube.com/watch?v=uOSnQM1Zu4w
Excellent video, thank you.
Two days ago someone cut a whole overland power pylon in Germany…
Link please?
In German: https://www.swr.de/swraktuell/rheinland-pfalz/strommast-abge...
I hope somebody building and selling microgrids and energy storage systems is ready to take advantage of this situation. Mass deployment of batteries at the grid edge is a great idea for a lot of reasons, but none more immediately compelling than "the power won't go out if the grid goes down."
It should be mandatory for critical infrastructure like ATMs to have 24hr batteries installed. And some sort of power backup to maintain internet access as well - even if it's just a matter of everyone being allocated a small amount of starlink data or public wifi.
These things have relatively small costs, but make the system much more resilient.
What do you do when the ATMs run out of cash? It's not an unlimited supply in them.
You should probably just keep some cash on hand. And keep an FM radio and a few batteries on hand.
My fear is that without access to cash, people might loot stores or riot to get essentials, once they get desperate. Keeping ATMs online for longer would make that kind of event less likely.
For those not aware how interconnected the European Grid is: https://storpower.co.uk/the-day-europes-power-grid-came-clos...
Living in Maine makes you prepared for this type of stuff.
It happens every winter. When I was a kid, we once went 2 weeks without electricity.
Wood stove becomes an obvious necessity. Not just for staying warm. For cooking.
Chickens in the backyard too.
Those moments show you that being self-sufficient is an extremely important skill, and while we're not totally self sufficient in those moments, we get pretty darn close.
Really makes you appreciate the pre-electric era.
People cut down forests with axes. Pulled the stumps out with oxen. Cut the lumber with hand saws. Chiseled foundations with pickaxes. Etc, etc.
With the constant threat of Nuclear War, there's a sword of Damocles hanging over our heads that we will return to this era.
That makes it all the more important to learn these skills.
It's fascinating to see simulations of power grid disturbances propagating due to the limited speed of light/causality: https://youtu.be/5vAkdXRyKWM?t=713
This is really interesting. Even though I'm used to reaching websites on the other side of the continent in less than a second, it's incredible seeing how fast a frequency disturbance propagates.
We had power on the canary islands (belong to spain, but located in the atlantic ocean ~80km west of Africa) since it is an independent grid. But couple of hours into the blackout (starting 5-6pm local time), internet issues occured - with confirmed multiple landline providers going offline (Digi, Movistar) while others seems to continue to work (Vodafone). Also some cellular 5G were affected, otheres continued. Interested to see the postmorten - I suspect after 8-10h the data center's diesel generators on the peninsula that route the traffic ran empty. Internet operation resumed during the night (for me around 1am).
First episode of Connections is great to watch on blackouts and technological dependence https://www.youtube.com/watch?v=XetplHcM7aQ
I've lost power at 11:30 and regained at 00:12 (Portugal). My hat's off to those brave engineers and technicians that managed to black start the ancient, under-invested grid in record time. Massive respect.
Mobile network is fully gone on the islands now — it was working until ~10 minutes ago.
Still working in mainland Portugal but it's getting slow, probably severely overloaded.
Yeah, it's back and slow. Sidenote: RTP (National Public broadcaster) is gone.
RTP came back for me an hour or so ago.
Regardless of the source of this specific issue, I suspect investment in infrastructure security and resilience is about to ramp up.
Calling attention to how fragile many of our critical systems are is almost certanly a net-positive in the long run.
I'm wondering: are there any simulator games in which you have to take care of the stability of a nation-wide power grid? This whole situation has me interested in logistics of power distribution now.
UK NESO’s website has one. It’s more educational that a real game…
https://www.neso.energy/energy-101/balancing-grid-interactiv...
The last major European blackout was in 2006:
https://en.wikipedia.org/wiki/2006_European_blackout
Confirmed by contacts on the ground in Spain and Portugal.
Not the best news source, but it’s the only one I’ve found so far. HN moderators, feel free to replace it later with a better one.
Supposedly also France is affected (unconfirmed)
This is the first brief article from El País, a leading newspaper in Spain: https://elpais.com/economia/2025-04-28/apagon-electrico-masi...
Translated version: https://elpais-com.translate.goog/economia/2025-04-28/apagon...
I have family in Andalusia and they seem to have dropped off the net. Mildly concerning.
Also, there are currently four different submissions re this on the front page. I'd suggest we dont need any more.
Yup I confirm. Went to turn on the car to hear the radio. Andorra seems to have power from French side, though in Catalunya we do not.
Should have paid the extra €€ to put the solar panels in backup mode…
I'm from Spain, electricity is almost fully restored in my area. But my fiber network is still down, same with the buildin elevators. People do not realise that when the backup supplies (batteries, diesel, whatever...) get drained, a cold start of non electric infrastructure could also be needed because syncornizing a mesh of unstable IT systems is tricky by itself, in some cases needing physical access.
EDIT:: typos
What network are you on? I'm outside Mataro and with Vodafone, got internet connection and power restored some time early morning/late night (after 02:00, before 09:00)
MasMovil / Orange. Yesterday connection went on and off while the router powered by and UPS. But since last night, the fiber does not sync with the GPON anymore.
> the fiber does not sync with the GPON anymore
Ouch, that sucks. Bet their technicians are on overtime already too :/ I'm guess you've already tried the true-and-tested "Turn off, wait five minutes, turn on again"?
Usually we've had lots of issues with our fiber from Vodafone (like random ~70% packet drops from time to time) but happy Vodafone seems to have recovered quickly in this case, no GPON issues here as far as I can tell.
I'm with O2 (Movistar) in BCN and the internet came back at the same time as the power at around 18:00.
Thats right around when we got phone service again, but no electricity until early this morning. Guess our internet/cellphone coverage comes from BCN which would make sense.
It's surprising how vulnerable electric supply can be.
Power back at 20:30 approx in some Madrid province locations at this point most of Madrid should have power.
The privatization of infrastructure in Europe has been a disaster. The only area where at least prices went down are telecommunication companies.
The postal services are worse now. Railway is more expensive and a disaster. Energy prices are sky high and apparently now we also see unreliability.
Can't you just wait for a minute 'til we know what happened to point fingers? sigh
This is a really good stress test. I’m impressed that communications still seem to work somewhat for now.
Yeah we only have telecoms and unreliable at times. If power doesnt comeback soon that too will fail.
Live view of the grid status, phase, and frequency as everything comes back online: https://gridradar.net/en/wide-area-monitoring-system
CNN story: https://edition.cnn.com/2025/04/28/europe/spain-portugal-pow...
In contrary to the CNN headline, air traffic doesn‘t seem to be affected. (As one would expect, as airports all have backup power systems)
Except Heathrow apparently
As of 2 min ago..( 14:32 WEST ) in many parts of Portugal electricity still out, and also Water systems also offline. Report is for some it can take up to 10 to 12 hours before power back up. Probably many locals not reporting here as they are low on battery.
Apparently Italy is also affected.
https://www.elconfidencial.com/espana/2025-04-28/directo-cor...
Outage map is down but no blackouts here (NE Italy) atm...
I’m in NW Italy and all seems fine
Tuscany good so far
Edit
However, I can't get the energy provided outage map to load, maybe too many people accessing it
I am getting downvoted for mentioning something from the linked article. Fun.
No
This outage does not affect the islands, at least in Portugal. Madeira has power.
Spanish islands are also all ok, they have independent networks
I’m close to Valencia and we’ve had no electricity for about an hour.
How can we follow this story? Who's updating in real time?
BBC https://www.bbc.com/news/live/c9wpq8xrvd9t
The Guardian https://www.theguardian.com/world/2025/apr/28/spain-portugal...
Knowing what I know about powersystems, it's a miracle electricity ever works and the people who manage it are silent heros. Please buy them a drink
For offline messaging over Bluetooth/Wi-Fi without internet or central infrastructure, I saw these recommended by ChatGPT Deep research and they seem current: Bridgefy (Android & iOS) – direct Bluetooth mesh within ~100 m, instant one-to-one & group chat, <1 min install. Berty (Android & iOS) – E2E-encrypted over Bluetooth/Wi-Fi; Briar (Android) – E2E-encrypted over Bluetooth/Wi-Fi with Tor fallback. Manyverse (Android, iOS, Desktop) – offline-first social feed that syncs posts over Bluetooth/Wi-Fi when peers meet. Can anyone vouch for these or know better?
Whoever downvoted, a comment explaining why would be nice, when the cell tower generators are out, and there is only phone battery left, offline communication options could be useful, I'm glad I'm not in that position right now.
Maybe because you don’t have to ask ChatGPT for what could have been a simple web search.
When I Googled it, the apps listed include Firechat, Signal and Briar. Signal is centralised, Firechat is dead, and I don't think Briar are current either? I wish it was a simple web search. Even Deep Research needed the ML pot to be mixed around a few times until it gave an good answer.
Seems like some people are not interested in honest and useful feedback because this got downvoted too! I thought it made clear points about an important topic. Well, whoever sees fit to do this, I hope you get whatever it is you want.
I'm in Portugal. Mobile data was slow when it started, now it's much better.
The sunny weather is very inviting outside for someone with the day off :-)
grid ops discussion on reddit about it https://www.reddit.com/r/Grid_Ops/comments/1k9uov6/major_out...
Wild that cloudflare was first to report this. At least first I saw.
I guess news networks don’t have comparable access to live metrics
I've heard from friends in France, Portugal and I am in Spain. I saw people are also mentioning parts of Italy.
AC is overly complex and too reactive.
I can confirm (I'm from Portugal)
They just reported on TV it will be on again in 20min earliest (15:30 CEST)
Anyone read Blackout[1]? So it begins. Both countries completed (or almost completed) their smart meter rollout…
[1]: https://en.wikipedia.org/wiki/Blackout_(Elsberg_novel)
Getting two network bars in Madeira. Seems to be affecting 4G/5G now.
Grady over on Practical Engineering is going to be so excited (-;
Same happened in Murcia. Just thought it was a normal power outage!!
My experience of it in Lisbon was undramatic. I was able to buy medication from the pharmacy. Life continued as usual more or less apart from some inconveniences, people playing cards in the street and a sense of everyone stepping away from the tech. Eyes were rolled as the reflexive suggestions of "putin" were inevitably voiced, more to make conversation than anything else. I met neighbours that I'd never talked to before, and reconnected with some I hadn't talked to for years. People played guitar on their balconies rather than spotify bluetooth speaker pop. The neighbourhood cheered as it came back online but part of me, selfishly, would have liked a few more hours of blackout.
Would have been a very different story if it were a week rather than a day, but I was left with a sense that this complex community of locals and foreigners is stronger than I previously suspected.
> The Portuguese electricity operator earlier said the outage was caused by a "rare" atmospheric phenomenon, related to variations in temperatures
Global warming/climate change strikes again
barcelona has been out for the past 20 minutes or so
Practical Engineer YT religious watcher here.
Power is back on some parts of Lisbon
Got reports from Toledo and Madrid
Sounds like China 30 years ago
i had an appointment in Barcelona, but can only guess it’s not safe on the roads
I just walk through Eixample, traffic is a bit chaotic and police has trouble managing it. Stay safe!
Traffic lights are out across Iberia apparently, better stay home.
Do people forget how to drive if there are no lights? I can't imagine many people would think "well, there are no lights, which means GREEEEEEEN!". Over here, a traffic light that doesn't work means it's now a stop sign.
That's still a recipe for massive gridlock as treating most traffic junctions as stop signs doesn't scale unless there are traffic cops to manually direct traffic
I've never seen a traffic cop but I've driven past maybe fifty or a hundred broken or shutoff traffic lights, mainly in our largest city and never got stuck. Local law states that you give cars coming from the right precedence and the crossing self organises around that.
It won't be fast, but it shouldn't really be dangerous.
Having driven in Madrid when lights were working, I think the stay home advice is strong.
People routinely forget how to drive when the traffic lights ARE working.
I work in Barcelona and live in a town 20Km outside it. I drove back home just fine at around 16PM. Some traffic in the city, but manegable. My town has had electricity in homes since 15PM, some neighbourghs told me. At night I drove to my parent's (different town, 10km away) since I couldn't communicate with them and was worried. Their town was pitch black. People were driving mostly safe and slowly. Of course you get the random POS that doesn't care, but I didn't feel in danger at any moment. My parents were fine. I drove back home later and same story. I just woke up (it's 7AM here) and the Internet is back and I have a message from my parents, their electricity is back too. Overall, I didn't see any crazy behaviour.
Maybe the grid upgrade should be to DC to get something hopefully more stable?
Yes, it’s happening
Carrington event?
There would be a broader area affected.
Murcia is also without power. We live in the campo so presumed it was a normal outage. Obviously not
[flagged]
This complaint needs to be passed to European regulators at either BEREC, ACER or DG Connect, since using a loginwalled site is a violation of EECC directive 2018/1972. Ideally they can just designate 1 approved site that all utilities should harmonize on using for their status updates.
Is this a joke? Because that seems like exactly the wrong thing, and such a system would itself be a bottleneck for downtime issues.
Using twitter has the huge advantage that spikes in users in Spain for checking this stuff is a rounding error in the normal traffic so is very unlikely to take down the status page.
no one would/should prevent companies from using also private channels like X, FB, Instagram, etc but enforcing a public channel that doesn't require private citizens to register, accept T&S and share their private data with 3rd party, unrelated private corporations to be informed of critical, public safety information would be helpful.
Aside from the login issue, Twitter has outages too. It really isn't really that hard to replicate posts across multiple services in 2025.
They may not be aware. We had Dutch fire depts do the same, it used to be possible to see tweets without logging in.
[flagged]
[dead]
It's not Musk fetishism. It's network effects. Most people hate X and hated Twitter but people use them because people use them.
This should show people just how powerful network effects are. They are legitimately a force of nature.
I strongly agree, and to me this supports the argument for heavier regulation (or any really)
[flagged]
Saying things you know to be untrue will not win you political friends in the long run; quite the opposite: it’ll cast doubt on every other opinion you have
That's funny, Fascism has never struggled to gain followers while lying about everything. In fact, an explicit abandonment of truth and reality is inherent in most Fascist regimes, and is also prominent in other types of cult of personality authoritarianism.
Maybe outright lies are more effective at getting followers than you suggest.
That being a distasteful reality doesn't make it incorrect.
There will always be a number of loons willing to support any fringe idea you might mention.
The key is: will the bulk of the people, who are in the center, go for it?
And the answer is: if they perceive the idea as being cynical, or purposely deceptive, then no. Even if politicians temporarily get away with lies, every lie does long-term damage to a movement.
The truth will out.
Only for those who don't understand sarcasm or other forms of figure of speech and art in general. For those I'll prefer distance anyway, so it's a win-win.
It's hyperbole, not sarcasm, and it's an effective way to neuter yourself politically. Have you ever rolled your eyes at someone going on and on about communists in the deep state?
Ok thanks for the heads up.
> Have you ever rolled your eyes at someone going on and on about communists in the deep state?
Not really, I'm indifferent to those that I'm not interested in what they have to say.
They did post on BlueSky, but not every update so I assume someone is cut and pasting manually...
https://bsky.app/profile/redelectricare.bsky.social
While I don't particularly like twitter (X), in a situation like this it probably has a better reach than the website of REE.
That said, twitter should allow for official profiles and organizations to have their tweets (xs?) made public.
You’re kidding right? The overwhelming majority of people do NOT use X or have accounts there. We are in a bubble. (Not making a Musk point here as it was the same before the “exodus”)
They probably did not patch their firmware in years and have their SCADA systems live on Shodan with default passwords...allegedly...
I remember years ago someone scanned the Internet IPv4 space for open unpassworded VNC servers. Many of them looked disturbingly like industrial control systems.
Just take a look at https://infosec.exchange/@shodansafari. A lot still look like that.
Even the water control system for a big French Dam:
"115 batshit stupid things you can put on the internet in as fast as I can go by Dan Tentler" - https://youtu.be/hMtu7vV_HmY
[flagged]
[flagged]
They could definitely just post updates on their website, the infrastructure is there, running, if they want to inform people on Twitter it's just a bonus to message it through the network that is Twitter.
But important/urgent updates only via Twitter is definitely a huge no-no.
[flagged]
I assume that a major electric provider has the capacity to build a simple status update system where they post. It's much simpler than any "blog platform" case study used to assess junior engineers. You are assuming that it's their IT systems which crashed the electrical network and there's no sign of that at all at this moment, you are misleading in there to try to score a point, stop.
It's far easier to use Twitter but it doesn't mean it should be used, it fences out people like the OP and me, who do not have Twitter nor want to have Twitter, I don't want to be forced into using a private corporation service to get status updates from the electrical network where I live in. It's quite a simple proposition and very reasonable, not sure why you are so incensed by a quite reasonable expectation.
> Also, why do you assume that website wouldn't crash under the sudden 10000x load? It is an utterly useless solution, that wastes time and solves nothing.
Because it can be cached very easily, it's 2025 where setting up this kind of cache is extremely easy compared to 2005.
> Like does your engineering skills suddenly magically evaporate the moment Elon's name is mentioned?
Please, stop, you are too irrational to understand a very simple and reasonable thing, no need to start throwing Elon into this bullshit, just stop here with the rabid lunacy. I was against major corporations only posting updates on Twitter waaaay before Elon bought it, I still stand by it.
> "I currently don't have any internet service and just €15 in my wallet - I can't withdraw any money from the ATM," she added.
from: https://news.sky.com/story/large-parts-of-spain-and-portugal...
This is literally the whistleblowers about cashless society have been warning everyone about for well over a decade now.
> whistleblowers
Using the term whistleblower in this manner is inappropriate; actual whistleblowers are individuals who bring to light illicit acts by organizations or governments at great personal risk.
Prophets
Peters
I'm a dummy, but I don't get it :) can you explain?
> This is literally the whistleblowers about cashless society have been warning everyone about for well over a decade now.
This is how humans are with all catastrophes–there isn't enough money until after something really, really bad happens and suddenly there is enough money to fix the issue.
NYC is extremely vulnerable to a 9/11 style attack on the fresh water aquaducts. Fuller wrote about this all the way back in the 60s in Operating Manual For Spaceship Earth:
Thus under lethal emergencies vast new magnitudes of wealth come mysteriously into effective operation. We don’t seem to be able to afford to do peacefully the logical things we say we ought to be doing to forestall warring-by producing enough to satisfy all the world needs. Under pressure we always find that we can afford to wage the wars brought about by the vital struggle of "have-nots" to share or take over the bounty of the "haves." Simply because it had seemed, theretofore, to cost too much to provide vital support of those "have- nots." The "haves" are thus forced in self-defense suddenly to articulate and realize productive wealth capabilities worth many times the amounts of monetary units they had known themselves to possess and, far more importantly, many times what it would have cost to give adequate economic support to the particular "have-nots" involved in the warring and, in fact, to all the world’s ’have-nots."
> This is literally the whistleblowers about cashless society have been warning everyone about for well over a decade now.
Indeed but it stands to reason that this outage will last maybe a few hours until the grid has recovered. A nationwide full blackout is a scenario that's on a "once a quarter century" level, and the last one in 2006 was resolved after two hours. It's Europe, not the US - our grids operate on much, much stricter requirements and audits on resiliency, hell since last year we got an active warzone in the ENTSO-E grid and it hasn't been too much of an issue!
Not much of value will have been lost in the meantime. The only ones who are truly and beyond screwed by such events are large smelters and similar factories where any prolonged downtime leads to solidification of the products which, in extreme cases, require a full reconstruction.
As for "I can't buy eggs in a supermarket now"... lol. People need to learn to chill down a bit. You won't die from having to wait a few hours to be able to buy the eggs.
> Not much of value will have been lost in the meantime. The only ones who are truly and beyond screwed by such events are large smelters and similar factories where any prolonged downtime leads to solidification of the products which, in extreme cases, require a full reconstruction.
I think you've left out a few things, I remember doing on site work at a pharma company that required some downtime on one of their lines and if we went over the allotted time, they would be charging us up to 2 million EUR an hour. Hospitals and critical services SHOULD have backup generators etc, but depending how long this lasts a lot of things can become a major problem.
The majority of the cases will be fine, but when there's mass confusion and interruption like this, there's always horrible stories that come out.
most generators in european cities are connected to gas utility so it has essentially unlimited capacity without land traffic.
edit: and europe has almmost always atleast half a year of whole country supply of natural gas in caverns and other storage.
> our grids operate on much, much stricter requirements and audits on resiliency,
None of that changes the difficulty of a black start. If there is a full outage, it will take a while to get going.
> None of that changes the difficulty of a black start.
That's the beauty of the European grid: it is not a black start event for Spain, at least as long as even a single link to any of the neighbouring countries is available.
The difficulty is still the same: for each piece of the puzzle (substations, high voltage power lines, transformers, generators, loads), they have to wait until enough power is available at the connection point, and carefully manage the power balance so it's neither too much nor too little (and if they get it wrong, things trip offline and they have to start all over), while the grid is in a degraded state (often meaning no alternative paths, so a single fault can set them back several steps). The only difference being that the black start "generators" are these still working international links (which could be very far from the important parts of the national grid).
It might be faster to instead black start several independent power islands in parallel, and connect them together as a final step. At least in my country (Brazil), that's how it's done for large-scale blackouts, even when some of the country still has power; it was done that way for the partial blackout in 2023, and there's a written procedure on how to do it (which is available on the operator website, if you know where to look). In 2023, some areas failed to black start for one reason or another, and had to wait for power from the outside; other areas managed to black start as expected, and were then synchronized with other areas until everything came back together.
Yes
But honestly dark starts are the kind of boomer self-made problems that'll just have to work around
Whoever built a solar grid inverter without the capacity for dark start needs a stern talking to
The grid is not just an aggregation of individual sources and sinks; it takes active effort to keep them all working together in a useful way without just collapsing again into another cascading failure. For that reason, your solar inverter doesn't come on until the grid operator wants the solar inverters to come on in your section of the grid.
It's tempting to think of the grid as something grid operators control, feeding power from point A to point B, but the grid is actually largely uncontrolled - the power just flows wherever it wants to - and the only controls they have are turning on and off generators, adjusting their throttle, disconnecting loads (rolling blackouts) and sometimes opening circuit breakers (though this is not normally useful). They don't even have precise real-time monitoring of the whole grid - only specific measurements in specific locations, from which the rest is estimated using lots of maths (which is how you would design it too, if measurement devices cost $100,000 apiece). That's why it's not a trivial task to keep it working.
However, you're able to have your own, private miniature grid, on which you can power your own loads from your own generators. It's even possible to do this with solar inverters! You will need to specifically seek out this capability, and get extra hardware installed, which is probably why you don't have it. You need a "transfer switch" to definitively disconnect your private grid from the main grid when you're using your private grid capability - it's not allowed (and not safe, and will blow up your equipment anyway if you force it) to just feed power onto your local unpowered section of the grid.
I knew some of this, but thanks for explaining it well
And while there are ways to maintain inertia https://www.renewableenergyworld.com/solar/grid-inertia-why-... I don't see why a solar farm can't do it through smart syncing of inverters (or maybe they do some measure of it)
Solar inverters could simulate inertia by not running at 100% most of the time, but you do want the free energy machines to run at 100% as much as they physically can, because it's free energy, which means there's no buffer for simulating inertia. It's been commented many times that batteries can be used to simulate inertia. You can also literally just add inertia, in heavy spinning lumps of metal that don't do anything.
As in the German government and most of the political class...
Germany has dozens of links to its neighbours. We don't need much in terms of "black start" capacity, that's just pointless fearmongering by fossil fuel and/or prepper propaganda sites.
As long as even a single link to any of our neighbours is up and running, it can be used to start the rest of the grid - which is exactly what was done in the 2006 outage and why that one took barely two hours to be resolved. The only truly screwed country at the moment is Portugal because all their grid links run through Spain.
As long as we’re throwing shade at EU vs US, I can’t remember the last time the US had a nationwide blackout, certainly not in my lifetime!
> As long as we’re throwing shade at EU vs US, I can’t remember the last time the US had a nationwide blackout, certainly not in my lifetime!
Talking about "national" in the sense Spain (pop. 48M, 506,030 km²) is roughly equivalent to a few US states. A similarly (population/area) sized outage occurred a couple of decades ago:
* https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
North America is organized in regional grids:
* https://en.wikipedia.org/wiki/North_American_power_transmiss...
US nationwide would be the whole of Europe category, which, I don't believe had ever happened.
Texas, on the other hand, which is easily the size of a country...
Just verified: Texas is larger than Spain and Portugal combined!
Indeed, and that’s the main problem: you can’t remember or know anything.
It’s is a known fact that in general the US power grid is orders of magnitude less reliable than in Europe. And the excuse of “the weather is more extreme” is just that: a lame excuse.
Just count the number of American households that have generators and/or batteries vs the Europeans if you really have an honest desire to know anything about anything.
The US has three “independent” grids so losing them all would be hard. But I believe at times Texas has gotten close, and East went pretty dark at some point recently.
CA of course has rolling blackouts for other reasons.
2021 was pretty bad in Texas, IIRC
2021 would have been a non-event if people in Texas weren't propagandized about some nonsensical "Independence" bullshit.
A few more interconnects with the rest of the country and it wouldn't have even made the news.
this is after decades of Texans bragging about their independent power supply. Many Texans still believe outright lies about the blackout, like it being "caused" by green energy sources, which was false.
It was caused by free market participants not spending capital to harden their network. Solar panels and Wind Turbines work great in the cold climate of Canada.
The storm that caused such a problem is a once every ten years storm. The grid companies all should have foreseen this with even minimal investment in planning. They didn't, because that's less profitable, and the "regulator" in Texas has no ability to punish them for pinching pennies on reliability and resilience.
Free Market at work baby!
>> The storm that caused such a problem is a once every ten years storm.
This is incorrect. That storm set multiple records, most notably the longest freezing streak the state has ever experienced [1]
Houston, San Antonio, Austin and Waco hit 30 year lows while Dallas set 80 year lows.
It also hit the entire state at the same time.
Maybe there's validity to some of the rest of your post, but that storm was absolutely not a regular occurance.
1. https://www.ncei.noaa.gov/access/monitoring/streaks/mapping/...
There's five different grids in North America (Eastern, Western, Texas, Alaska, Quebec) so something would have to go very wrong for a nationwide blackout.
Except this is Spain and Portugal which combined have 1/3 of the area of Texas, which had a state wide outage recently.
There was a very famous one in 2003.
As was pointed out, the USA has three independent grids (east, west, and Texas) and EU countries are roughly comparable to states (except with less federal power). The equivalent of a European nationwide blackout would be a US statewide blackout, and those HAVE happened, definitely within your lifetime if you're old enough to use Hacker News, mostly in Texas.
It was about 7 hours long: https://en.wikipedia.org/wiki/Northeast_blackout_of_2003. Where I lived it was restored very quickly.
I had a long blackout as a kid during a hurricane in 1985. Once it was safe it was repaired rather quickly.
US, no, Texas and California are .. not doing so well.
There isn’t much you can buy with cash either if there is no electricity.
I wonder if stores would be able to sell anything without power. My supermarket, for example, only has electronic cash registers. And no price tags.
I was on a plane during the Northeast Blackout of 2003. Landed, got in my car, and attempted to leave the parking garage but with no power the automated EZ-Pass payment wasn't working nor were the credit card machines. Most people, myself included, had neither sufficient cash nor a checkbook on hand. Huge logjam of cars. The workers ended up getting the old school handheld metal credit card machines that created an imprint of the credit card on carbon paper from some long forgotten storeroom and using them.
Many years ago I worked in a Safeway grocery store. We would have occasional power failures that would leave the entire store dark; we would all be given flashlights to help customers find their way out.
The cash registers, though, had backup power, so the store could still take their money.
I worked as a cashier at a large UK supermarket when I was a student. In our training we were told that if there was an outage with the cash registers we should ask shoppers to estimate how much their groceries cost and accept what they told us. Payment could be by cash or cheque.
Apparently when this had been done in the past shoppers were generally honest & relatively accurate.
Many situations call for pen-and-paper backups. Giving out receipts the old way should, in theory, be a possibility, then backfill the computer system later.
Sure, but if we are talking about backup in outages you can also get credit card payments when power is down using something like a Stripe Terminal.
This is actually exactly the case that I had in one trip to Andorra: the power was down for 2 hours while we were choosing equipament for skiing. The shop had no issues getting our orders done though, because they just manually filled the orders with pen-and-paper and did the payment with a credit card terminal connected to a smartphone.
If your building has a power outage, that works.
If your city has an extended power outage, the cell nets could easily be down as well.
In my experience, for the 4G/5G network to be down something really serious must be happening. I had long power outages (more than a few hours, in some cases even days) that affected multiple regions in places that I lived before that still had working cell network. I assume cell networks have backup power and preferential usage of the power grid, but I am not a specialist.
And I am not saying that you shouldn't accept money as backup, of course you should. But what I am saying is that you can still accept credit cards even during most power outages.
Same as Software Engineer, it is impossible to have perfect, 100% reliability, but it doesn't mean we can't improve from 99% to 99.9%, for example, to have a better service.
Used to work for a telco. Cell sites have battery backup. Some have generators. Any fibre repeaters also will, as do any radio based backhaul sites. The HLR/core network etc will run indefinitely due to generators & strict fuel supply contracts for said generators.
If a city has an extended power outage such that the battery backed cell network goes down, then everything else will be failing too and payments are the least of your problems.
Without electricity the water system depressurized, which contaminates it. After about a week the sewage pumping stations have backed up so the sewer system is starting to fail.
Modern cities cannot operate without electrical power given their scale and density.
It is bizarre to think the biggest problem is "how do we keep a transaction of value?"
Like, just declare an emergency and let business owners be reimbursed by the government.
Credit card processing existed before widespread telecommunications infrastructure. Maybe we should require payment cards to have raised numbers like they used to so the old carbon copy machines continue to work.
Credit cards and payment networks have always explicitly supported "Offline" processing like that.
The kind of fraud that system enables isn't really common.
In Lisbon's airport they are temporarily back to just stamping the passports without the biometric stuff. There are already reports that the checks are being lax.
I was in a Tesco when there was a power outage - the self checkouts all died (and rebooted into a custon Debian based OS with a Chromium front end by the looks of it), but the staffed tills still worked and could accept Chip+PIN but not contactless.
> My supermarket, for example, only has electronic cash registers. And no price tags.
I know someone who works at a supermarket, and (some of?) their point of sale (POS) systems have a small UPS that can run for a couple of hours to ride through smaller outages.
I used to install those systems, and it's a couple minutes at best, if they've replaced the batteries recently, which they never have. It'll give them time to finish the transaction they're in the middle of ringing, then shut down cleanly, because the server in the back room _should_ have a bit more battery to keep the database consistent.
PoS systems aren't particularly power-hungry, but store owners never want to spend an extra cent, so they go with the smallest UPS they can manage. (And arguably if they went with a big overkill UPS, its after-outage recharging power would be larger so you'd be able to put fewer registers on a single circuit, so it's not as simple as just dropping in a bigger UPS.)
It's not just stores, pretty much our entire socity runs on cheap power, without it the whole thing falls apart.
Our global society runs on a lot of cheap sources of necessary inputs. Power is just one of them.
Very good point that nowadays many stores rely on barcodes and product IDs to get prices, and don't label individual items... So even pen and paper to keep track of takings is no use since they can't even figure out prices if the system is down!
Valid point, but there must be a price at the location of the item in the store.
> My supermarket, for example, only has electronic cash registers.
That's insane to me, in the EU anyway it's not permitted to only accept electronic payments..
> Retailers cannot refuse cash payments unless both parties have agreed to use a different means of payment. Displaying a label or posters indicating that the retailer refuses payments in cash, or payments made in certain banknote denominations, is not enough.
>That's insane to me, in the EU anyway it's not permitted to only accept electronic payments..
That's not the case. There are individual laws in each country that govern this.
https://fullfact.org/online/UK-not-only-europe-country-legal...
I guess I was taking the recommendation as a regulation, my mistake.
Either way, there should be no reason grocery stores don't accept cash imo.
https://www.ecb.europa.eu/euro/cash_strategy/faqs/html/index...
They have to accept cash in the US as well. The post you're replying to is just saying that they can't ring anything up or accept any payment without power.
> They have to accept cash in the US as well.
Only in a handful of cities and states. There is no federal law requiring businesses to accept cash for goods and services.
Ah okay, that wasn't clear to me, I imagined only being able to use electronic payment types.
But in this case, an emergency, I would assume someone would still know how to take a manual payment receipt!
And this is a trial for how a Carrington event will look like.
I just look to Gaza strip as an example how society continues operating under extreme adversity from the western powers, incl. total lack of electricity distribution for more than a year and a half and full blockade of water, food, and medicine.
No need for imaginary scenarios.
Eg. https://www.dropsitenews.com/p/palestinians-gaza-create-surv...
You mean the part that anything above 130nm technology would be fried?
I don't think this is true, and I would like to see an explanation of how it might be that quantifies the effect: are we talking electric field strength or magnetic field strength? Modern parts are generally better at ESD than old ones, as well.
Why 130nm?
Yes. For the moment I just want to see for how long the AWS Spain Region will stay up...
Unless this extends to many days, the diesel generators will cover this easily.
most datacenters do not use diesel, but gas/methane/natural gas - utility connection, that way you do not need to truck/store anything.
and so being located in middle of city you do not want ten of thousands of liters of diesel in tanks there.
same applies for luxury high rises in europe, almost all if not all of 20+ story buildings built last 30 years have them on roofs.
> and so being located in middle of city you do not want ten of thousands of liters of diesel in tanks there.
Why? Every gas station stores that capacity.
gas stations, at least in europe, have "safety zone" around them.
also depends on data center, some datacenters are directly in middle of european city. in most dense part of city and dense in europe or asia is something else then dense in america.
This isn’t true in the US and would be a huge dependency risk. Natural gas networks require electricity to operate.
> and so being located in middle of city you do not want ten of thousands of liters of diesel in tanks there.
This is based on FUD. This is the case all over the US and they don’t cause problems.
It’s interesting that Europe has taken such a brittle approach to infrastructure.
Another reason why itcoin is a pipedream too, particularly for third world countries
If an ATM is down, it's down. With Bitcoin you can still use mesh networks. Here's an old article about an example: https://bitcoinmagazine.com/culture/samourai-and-gotenna-ena...
(Samurai wallet is defunct, but the principle holds up)
I agree that bitcoin is a poor solution for anyone but a small elite. But as long as you have the mobile network on diesel generators (which you do in countries where power cuts happen regularly), something like M-PESA still works.
You can have physical bitcoins - devices that hold a private key in a tamper free way. The private key holds a fixed amount of BTC
Without power or an internet connection you can't validate that the btc hash is unspent, avoid double spends, or validate that your taking ownership of it made it to the blockchain and was confirmed for a few blocks.
Sure you can transfer the private key from one device to another, but (a) you can't know the other person didn't retain a copy of it and (b) you would be limited to spending the exact amount you have in an existing transaction because you couldn't send a transaction to the chain that splits it.
You've to got a reputation system with identities also stored on the blockchain, and you accept the transactions only from the most trustworthy people. Counterfeit paper currency exists, and counterfeit bitcoin (not BTC) transactions might exist, but not every person is equally suspect of using counterfeit money.
Sure, but then you aren't really using bitcoin. The whole point of bitcoin was to be a trust less system where transactions are verified cryptographically on the blockchain.
Transacting bitcoin private keys without going to the network and trusting the other party to not scam you defeats the whole purpose.
[flagged]
If anything, it shows how little use there is for physical cash , only as emergency backup
"emergency backup" is a very strong, rational use, and should never be neglected.
You can use cash but it mostly does not work in case of big chains or stores. They need to have access to their SAP/ERP software...It is not about payment for a long time.
Running a generator to power a point of sale machine and maybe an offsite server is a lot easier than generators to power the whole electronic payment chain.
Cash is not really relevant to this particular discussion.
When the power is out one cannot pay with cash either - because the cash register is offline.
Let me introduce you to the physical receipt block which is made of paper and can be filled up with a pen, and what is still often used in, for example, Christmas markets.
Depending on the country it’s not that easy. Maybe something changed in the meantime but where I am those blocks are prefilled-numbered and stock needs to be controlled.
It’s not like you can (could?) keep a block „just in case” and thus many shopkeepers wouldn’t even bother in case of outages.
Depending where you live a good old trust can be a currency. Humans are great when it comes to adaptation, I bet I could just write on paper name, CC number and leave it on a paper for shopkeeper and everything would resolve just fine..
I hear you, and in my second job 20+ years ago we were still trained (in a cursory fashion) to use physical charge slips in case of a POS system network outage, a full power outage would strain my imagination for ability to transact business at the majority of stores other than small mom and pop operations, and tbh even if such an outage had happened to me in 2005, I would not expect my department store to remain open and conduct sales.
Most retail workers are GenZ and struggle to understand what this would even look like because they’ve never conducted any transaction without POS computers (for looking up prices, for tallying them, for figuring tax and total, and computing change), so even if a dusty manual in the stockroom technically spells out a method of ringing sales using nothing but pen and paper and maybe a solar calculator, I would be surprised if any of the clerks working any given day would have the initiative to initiate an offline protocol. Most likely the store manager would usher customers out, lock up the store, keep the staff for 30 minutes to see if it came back on, and then go home.
How do you open the cash register for change when power is out?
> How do you open the cash register for change when power is out?
I've only seen a few but I believe they have springs on the inside and roll on little wheels similar to how desk draws roll. Most can be opened with a key to trigger that event.
With the physical key, everything doesn't need to be electronic you know
Many countries require receipts to be sent to the tax services instantly. In many cases a long enough power cut (days) would render all transactions illegal
(And in many cases you cannot legally pay large amounts of money in cash, it has to be electronic)
Which countries exactly? I've traveled in UK and France and in both countries when the online cash register was down, they opened their physical ledger, made an entry and gave me a physical handwritten paper receipt written with pen and ink. They said they would make enter the same data online when the online cash register comes back up.
I believe this is France (NF525), but I don't think it says "instantly".
My understanding is that this (NF525) only applies to computerised cash registers (software must be certified NF525 compliant), which there is no obligation to use in the first place.
So it is perfectly legal to use pen and paper and a cash box.
Most governments are not as stupid as the anti-government wingnuts would have you believe. They will not prosecute you for taking transactions non-electronically during a power outage.
I'm sure they'd make an exception.
How can this even be a legal requirement?
In some countries whenever you print a receipt, a copy is also sent to the IRS equivalent of that country. Obviously there are events where that can't happen due to technical reasons outside of the store's control.
Which countries? And, again, I doubt that this is the full picture because there are many cases where people simply don't "print a receipt" perfectly legally...
Germany for example mandates printing a receipt. The receipt must be stored in a certified storage inside the cash register and is signed cryptographically, including the hash of the previous receipt such that there is a hash-chain of printed receipts. Therefore each printed receipt that the customer takes home (and maybe at some point hands in to the tax office for some reason) can be used to check the integrity of the cash register storage and all prior receipts in the chain.
https://www.lexware.de/wissen/buchhaltung-finanzen/neue-rege... https://www.lexware.de/wissen/buchhaltung-finanzen/kassenbon...
Many other EU countries have similar regulations, and in some cases had them for a long time.
Same in Portugal. Sync with the tax authority can be immediate or deferred (every x days). Obviously, you can still invoice manually using a receipt book, in case of failure or unavailability of software systems.
Thanks! European red tape madness strikes again... At least in France cash registers are not mandatory (for now...) so there is a way around this madness.
This is a bit more than just red tape madness, it's a strategy to make businesses more transparent. This is about trying to reduce non-reported transactions and too many people dodging their reporting. Even if the rules for cash registries and reporting are detailed, a) that's not really expensive for businesses - it's easy to automate and there are quite a number of competitors; b) compared to accounting and tax rules, they are dead simple.
Receipts or invoices are the basis for a firm's whole economic activity, including the underpinning of their financial reporting, their tax burdens etc. And businesses failing to provide receipts erodes not only the tax base, but also any rights a consumer may have.
The thing with red tape madness is that it is always perfectly justified.
Its actually less red tape. Getting a second copy of a given invoice is trivial, processing of invoices for tax rebates is also mostly automatic (such as health and education expenses); tax invoicing uses well-defined formats, so its trivial to migrate between systems, and perform all kinds of analysis. Also, it increases transparency - you know that eg. the VAT you're paying is not ending in the vendor's pockets.
What red tape madness are you even talking about?
Force majeure should apply.
Source?
Despite the EU having a directive to not make electronic invoicing mandatory, they now allow exceptions that introduce mandatory online invoicing. For example greece romania italy: https://sovos.com/blog/vat/greece-mandatory-b2b-e-invoicing-...
In Italy it doesn't have to be in real time. It's a once a month thing!
Store owners just make out paper receipts in this case, like businesses used to do back in the day
The gas station I worked at used a paper bag when offline. Good, not great.
How did the pumps work when there was no power?
No power is different than no internet, so that was different. We were a 24/7 store (7-Eleven) and sometimes we would close if it was a prolonged power outage. No one has 100% uptime, not even 24 hour gas stations.
They used to have a crank!
please be more respectful to the manager of that establishment!
He said "offline" not "no power".
Also, fuel station can probably successfully run it's own backup power;)
They don’t, at least not here in the US. In areas with power outages, gasoline stations do not operate.
A cash register is not required to make cash transactions
Why would an employee tracking device be necessary to make transactions? It is just people without a cash position performing mental gymnastics in an attempt to justify it, by trying to bring it down to the level of electronic payment methods they convince themselves that, in a similar event, they would be no better off with cash, when in fact they almost certainly would. A merchant who refuses cash on account of a disabled cash register, under these circumstances, is not a serious one, since to do so would be to refuse all business. Those merchants who are so encumbered by scale and bureaucracy that they would be completely incapacitated should be circumvented.
Cashier scribbles furiously to add to cash register later
> When the power is out one cannot pay with cash either - because the cash register is offline.
Cash registers can be connected to small UPSes to ride through smaller outages. You wouldn't need a larger battery if all you want to do is ride through a few-hour outage, or even a whole business day (8-12 hours?).
[dead]
[dead]
Reported here https://news.ycombinator.com/item?id=43819791#43820127
We eventually merged all the threads.
Here too:
https://news.ycombinator.com/item?id=43819914
Call me crazy but I'd love to see full power outage at least once in my life
It was actually pretty cool to bike through the streets. It all felt so calm. Stores/restaurants closed, no traffic lights, little traffic.
Parks were full of people. Literally chilling, walking dogs. Most weren't on their phones either (no service).
I used to live in an area that regularly had power outages during thunderstorms. I was also once in London during a power cut that took out most of the city centre. It was a strange experience, but not as weird as walking through empty streets in the time of covid.
Maybe just my perception, but power outages seem to be getting rarer with time, though when they do happen they seem to be far larger.
We had one in my country when I was a kid 30 years ago after a substation tripped and took down the rest of the grid. We broke out the candles and chilled I guess. Power was back up in my area that same night.
Well, maybe you can see one now, depending on how far from Spain you live!
In Japan, sometimes earthquakes will cause regional power outages, though they are usually recovered quickly.
I was living in Chile during its 2010 earthquake (https://en.wikipedia.org/wiki/2010_Chile_earthquake). The power outage in my hometown lasted approximately two weeks.
Has not happened to me yet in Hong Kong, even through 2 Typhoon Signal 10s.
Hopefully there won't be any loss of life due to this event.
[flagged]
Spain and Portugal are quite big and with harsh terrain, I doubt anyone would bother invading in a traditional sense.
But we have been invaded countless times during our history, although the last time was Napoleon in 1808...
Who would even invade Spain and Portugal? E.T?
Well, we love all kinds of tourists. Why not ones that could change the world? :)
#irony
Morocco might (again).
That would be a hell of a plot twist
[flagged]
[flagged]
[flagged]
[flagged]
Maybe just flexing a bit, establishing dominance?
What beef does russia have against spain and portugal?
https://www.lamoncloa.gob.es/lang/en/presidente/news/Paginas...
Parts of EU.
Ok...? Is there like more to this narrative or are we supposed to buy that russia just hates all of europe for being europe now? Whatever happened to the "rational actor" canard?
Russian establishment DOES hate western civilization, that is not "narrative" that is fact told by them in live tv to your face FOR LAST 70 YEARS. That is not "narrative".
Western style of life is not only EU but also USA. I do not know how people can even doubt this lol.
If they wanted western life in russia, then establishment will make changes to have it there, no ? Russia is NOT democracy, it is tyranny, autocracy. Again it is not narrative it is what they do there lol
clearly yandex translator is not good as gogole translator . ;)
Yeah yeah they hate it so much they send their children to live in Europe and buy properties there.
Hating the west is only an ideology given to plebs
I don't buy this argument, it's the pinnacle of selection bias.
`any != all` after all.
Your argument is essentially; because some Russian people send some of their children to be educated or buy some property in the west (as a portfolio of how many?) that the argument that the state of Russia dislikes the EU holds no water.
To me, it's hardly evidence of anything, just like how some people in the UK fetishise Russia- yet the UK government is actively hostile and condemns without hesitation- Russias actions towards Ukraine.
My argument is not just about "some people", people who are pretty high up in the hierarchy. How about russian ex-president?
The "hate west" narrative is pushed because it makes sense during the war. If Putin decides now praising the west will let him keep the power the propaganda machine will do a 180 turn
it is autocracy, not democracy.
so they HATE west no matter what they say, so you are correct in that.
but you are making wrong conclusion,
machine is not bad thing BUT they are good people.
They ARE bad actors no matter if they use propaganda machine that way or any other way or not use at all. they are bad actors period. propaganda machine is separate thing.
They hate west because by western standards they will be in jail for all LOOTING of russias natural resources and killing and abusing their own citizens. West is punishing people who do bad things. Russian oligarchy kids did nothing wrong, presumably.
contrary to west, in russia you get beaten by police because your children in west posted something on Xtwitter...
Alexei Navalny, Boris Nemtsov, Boris Berezovsky, Sergei Magnitsky, Stanislav Markelov, Anastasia Baburova, Natalia Estemirova, Anna Politkovskaya, yuri Shchekochikhin, Vasily Melnikov , Vladislav Avayev , Sergey Protosenya, Yevgeniy Palant, Yuri Voronov, Ravil Maganov, Vladimir Sungorkin, Anatoly Gerashchenko, Vadim Boyko, Vladimir Makei, Grigory Kochenov, Vladimir Bidenov + Pavel Antov, and thousands of others.
most spectacular was - Pyotr Kucherenko where two men holded him and third put shopping bag on his head, and noone saw nothing in whole plane... except three photos were taking of incident...
[flagged]
[flagged]
Europe is helping Ukraine it its defense against Russia. Russia has sabotaged a lot of things in multiple EU countries, including Spain. It's not far fetched to imagine Russia being the root cause of this, or being implicated in some way. Even if they are not, they 100% are watching this closely and learn how they can disrupt power throughout Europe.
How would this do anything but weaken the Russian position? The EU is clearly otherwise willing to watch Ukraine fall.
> The EU is clearly otherwise willing to watch Ukraine fall.
All the money, humanitarian aid, weapons, intelligence, training and geopolitical backing beg to differ.
Did Russia say they did it? No.
So you can weaken your opponent without getting the backlash.
[flagged]
I'm not saying Russia did it, just that they could be the cause of this. Weakening Europe is in their interest, and they already have blatantly done smaller scale sabotage. It wouldn't weaken their position, as you said, Europe is not really interested in taking Russia seriously atm. Sabotages, killings, politician corruption and public disinformation have been common tactics for them for a decade now. Now that Russia's #1 enemy is under their control, I'm not sure they are afraid to take on Europe more directly.
The EU is the main provider of financial support to Ukraine, the country which Russia is currently attempting to invade, along with one of the major provider of weapons and training.
[flagged]
Well, it's been for quite a while now. Acts of micro- (or mini-) aggression like fires, explosives in commercial shipping services, broken sea cables just became daily news.
The only positive aspect of this is after the root cause is found, the grid will become more resilient in the long term (but these kinds of changes typically take long time).
Again, how would these actions do anything but weaken Russia's position given the EU's apparent willingness to stay on the sidelines? Wouldn't ukraine benefit the most from the perception that Russia is at war with the EU?
For the record, I don't think that this particular event is a Russian act of sabotage.
But Russia is an aggressive authoritarian state that was already caught for (smaller) acts of sabotage in EU, some of them quite dangerous. Why they are doing this? Who knows, war in Ukraine was not rational too. Perhaps some people want to be evil just for the sake of being evil.
As a Russian emigrant, I long stopped trying to rationalize Kremlin decisions. Why authoritarians are authoritarians? Who knows. Mad with power or something.
The strategy is not even a secret. Russia sees itself as a major influence of the Europe.
You cannot control stable governments, so you destabilise them with various tools for prolonged periods of time and then you end up with a country which is much easier to influence.
How could this weaken Russia's position without a smoking gun pointing at them?
Same with the undersea cables.
Outside of discussing whether Russia is behind this or not, the broader Russian strategy seems aimed at undermining trust in European governments. [1]
The goal would be to create enough pressure from people - frustrated by problems like power cuts — so that governments must withdraw their support for Ukraine.
Any "WW III" fearmongering is similar : intimidate everyone into withdrawing support.
Many European countries have created emergency guides to help citizens preparing for crisis like this one. [2] This, I guess, has the underlying goal of maintaining trust in European governments.
[1] : https://www.politico.eu/article/russia-increasing-hybrid-att...
[2] : https://www.reuters.com/world/europe/eu-commission-urges-sto...
> given the EU's apparent willingness to stay on the sidelines?
... Wait, how are you defining that? Much of the EU is about as close as it is possible to be to being at war with Russia without actually sending in troops.
Spain helped ukraine with planning of this - https://apnews.com/article/russia-moskalik-car-bomb-322ab66d...
so blackout is attack from russia. so stop spreading lies of terrorist russian state.
Reading this thread from Russia feels surreal.
Good place as any to ask: do you need a VPN to access the “Western” internet? Is the block on the Russian side, or are Western websites blocking Russian IPs?
No in most cases. Meta products are banned, twitter, discord and youtube (this one mostly works in reality), but pretty much everything else is unaffected.
Some Western side companies banned Russia by IP's like Intel, but in general, my list of websites to tunnel through a VPN is rather short, like a dozen and mostly to unblock youtube as meta and twitter are cancer anyway.
The block is on the Russian side (in most cases), but not all Western sites are blocked (Hacker News is working). Most of Russians know how to use VPNs, though it is extremely inconvenient.
We feel similar when watching this:
https://www.youtube.com/watch?v=0unhxkWkiKY
Why is that? We're not saying it's definitely Russia, but exploring the possibility they could be behind this.
After the multiple sabotages, killings, corruption, as well as the invasion of a neighbor country, we have some reasons to think Russia is a bad state actor.
[flagged]
Why?
I understand, but unfortunately, I don’t see any claims that are false.
not all people in west are that ignorant, atleast half of european public wants putin gone.
(other half is just living life not really caring about putin.)
[flagged]
Some more-unhinged ultranationalist elements of Russian societies loathe Western culture and what they see as 'decadent' Western values on their culture. This is not new. Xenophobia and hate of the Other has a long and sordid history in Russia.
Moreover, they are unable to just live-and-let-live and actively go out of their way to make other peoples lives miserable. This is due to pervasive zero-sum thinking in Russian strategic thinking. They are fixated on the idea that in order for Russia to 'win', others must suffer and lose.
Russia is an imperialist, terrorist state and wants to see all of Europe in its "sphere of influence"
They have already assassinated People in Spain last year
https://www.politico.eu/article/maxim-kuzminov-russia-ukrain...
[flagged]
Spain underinvests in its military. Russian mafias also control much of the costas.
I would surmise that the Russians think that Spain and Portugal are cowed, and want to keep them intimidated and prevent them from increasing their aid to Ukraine.
[flagged]
Some people close to power in Russia (Dugin) actually seem to believe Russia natural range is from Lisbon to Vladivostok.
"Putin channels ultranationalist discourse, such as the Izborsk Club and the neo-fascist Alexander Dugin, in calling for quasi-religious rebirth of Russian dominance, an agenda that seeks to swallow “Little Russia” into a renewed Russian empire that stretches from “Lisbon to Vladivostok,” a phrase popularized by Dugin and repeated by Putin."
https://brill.com/view/journals/joah/4/1-2/article-p126_10.x...
Dugin's views and influence is greatly exaggerated in Western media.
>renewed Russian empire that stretches from “Lisbon to Vladivostok,” a phrase popularized by Dugin and repeated by Putin."
This is a direct lie. Putin has never said this.
And one of the greatest lies that is being spread about Putin that he intends to conquer Europe and recreate Russian Empire.
I often have the same thoughts as you because, based on what I’ve seen, their actions aren’t strictly rational. For example, the damage to undersea cables will just be repaired, and it only end up angering everyone. Sometimes they also start local fires. I don’t really understand it either.
I do not really think that this needed to be a russians work tho. Spain and Portugal are really kinda far and it would be massively idiotic move even for them.
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
Noth Koera doesn't have any beef with Europe. Whereas the Russian propaganda says quite shocking things about Europe every now and again, not to mention their previous prime minister that is suggesting nuking European cities on Twitter.
North Korea fight in Ukraine, so they could simply be hacking for hire
[flagged]
[flagged]
Genuine question: how is it that Ukraine’s electricity grid still works to any degree despite being bombed for years and no doubt intentionally sabotaged / cyber attacked.
But a whole country’s grid can go down like this in an instant?
The answer is simple: EU and USA taxpayers billions pumped into Ukraine to buy hundreds of thousands of mobile diesel powered generators and fuel for them, and spare parts and replacement blocks for everything that’s been destroyed.
On top of that, Ukraine inherited a lot of nuclear power plants, and despite losing the largest one in Zaporozhie region, still operates all the other as Russia doesn’t attack them in any serious way.
No miracles, just pumping money non stop.
Maybe because it has not been privatized, yet?
https://en.m.wikipedia.org/wiki/Ukrenergo
[flagged]
-> Power outage -> Wild guess that Russia has been doing it, 0 proof or even hint -> "This is a war of agression, we should respond"
Are you suggesting to attack Russia, based on absolute thin air?
> thin air
Except for the mountain of evil, violent, underhanded and illegal stuff Russia keeps getting caught doing, and has so far escaped scot-free due to Western cowardice?
No, they were suggesting attacking it based on speculation about this specific event, so yes out of thin air.
I agree that Russia is evil, but saying that it got away with it Scot free is just insane. In fact it even plays into Russian propaganda, to say that nothing really harmed them even after the past 3 years of international sanctions and consequences.
And western cowardice? Apart from nuclear war (which I'm glad people are being "cowardly" about), what do you suggest? It's also funny when war mongers online talk about cowardice. Why dont you go volunteer in Ukraine if you're such a non coward?
> Apart from nuclear war
> It's also funny when war mongers online talk about cowardice
> Why dont you go volunteer in Ukraine if you're such a non coward?
This is an impressive feat. You've managed to package not one -- not two -- but THREE Russian propaganda talking points into a single paragraph. I believe I've just witnessed the second coming of Alexander Pushkin.
And you've managed to pull the typical online war mongers deflection of calling me a Russian shill. Let's say you're right, I'm a complete Russian shill (nevermind that I completely abhor the Russian regime since their intervention in Syria, and think that Putin belongs to the Hague tribunal).
Are you going to actually dismiss the fact that yes, nuclear war would be a very likely possibility if we declared war against Russia? Much more likely than... Not doing that?
Or are you going to use the Reddit argument that somehow, it just won't happen because trust me bro, and only Russian shill would care about getting nuclear bombed?
And yes, it's cowardly to call people cowards because they won't push for a war that you yourself won't fight. I didn't bring up cowardice first by the way, but I guess it's fine to do it when you agree with the person lol.
You seem pretty upset. Are you okay?
I'm not. But you are the one going around accusing people of being Russian shills, I hope your little red scare gets better and you move on from Redditor style arguments :)
[flagged]
As much as I am anti-Russia and against the invasion, this is absolutely not the place to start this flame war.
The thread I answered in had someone playing dumb as if there would be no reason whatsoever to consider Russia a hostile actor, I was responding to that.
That's not what you said though. And it's still a baseless accusation. It's similar to accusing Israel or the US of every single coup d'état or sabotage action just because they have a huge history of doing that. In both cases it's completely baseless unless there's evidence.
That's exactly what I said - Russia is a hostile actor that has given plenty of reasons for war, regardless if they're involved in this, which neither me nor anyone else knows.
War by who? Who do you think should declare war on Russia? And as another comment said, are you volunteering? You can even wage war against Russia now if you want to, volunteers in Ukraine are always welcome.
> War by who
By the international community of course. The same one that has heavily sanctioned Russia.
> And as another comment said, are you volunteering? You can even wage war against Russia now if you want to, volunteers in Ukraine are always welcome.
That's just a cheap deflection. Me dying in the trenches won't change anything; a coordinated properly aggressive response might teach the war criminals at the Kremlin that they will not be let off. It's just a repeat of Munich/Ethiopia, and no individual at our level can do anything about it but advocate for change in policy to our governments.
It's not a cheap deflection. You want other people to die in the trenches or worse, to die in a nuclear war. The deflection is to say that "your life won't change anything", which I mean sure yes, but that's what every single war mongers that doesn't want to fight the wars that they promote say. If everyone said that, no one would be fighting against Russia right now.
And it's funny to say that it's a repeat of Munich / Ethiopia. Again, war mongers will always prefer comparing to the few times where war was the only solution. The only issue is that not only do both situations have nothing in common (not that anyone cared about Ethiopia, since European colonialism in Africa was nothing new), but it also erases the fact that we are talking about nuclear powers here. If Hitler had nuclear weapons pointed at London and New York back in 1938, then obviously the equation wouldn't be as simple as "stop him now" even with hindsight lol.
> You want other people to die in the trenches or worse, to die in a nuclear war
Other people are dying, and more people will until the aggressor is stop. The aggressor has shown loud and clear he has no intention of stopping.
> Again, war mongers will always prefer comparing to the few times where war was the only solution
Because the situation is comparable. Putin has clear and overt ambitions for further expansion (one of his minions already published the plans with Moldova next). If he isn't stopped, like he wasn't in Georgia and Crimea, he'll just continue. So the comparison is very apt. Yes, nuclear weapons add a serious nee dimension to the problem. That doesn't mean we should let Putin get away with war crimes.
Edited to add: https://kyivindependent.com/russia-military-building-up-at-f...
It is quite common to see bad geopolitical actors paying people to go online and accuse Westerners of being "Warmongers" or telling people who object to Putin's wars of imperialist aggression to "go and enlist and fight" with the subtext that the audience are weak and decadent Westerners with no stomach to fight a literal Mafiya-run fascist dictatorship.
Speaking strictly for myself, given that I want to avoid having people think I'm apologising for the Putin regime, I do not ever use these arguments in good faith.
Nope, that's not what I said. I'm a westerner, and it's not about "good faith". It has nothing to do with behind decadent or lazy.
It's because the modus operandi of war mongers, since before world war 1 has always been to agitate for wars that they don't want to fight for themselves. We are talking about nuclear war here, the bad faith argument is to imply that being against it is a sign of being a paid Russian shill. I guess the Pentagon is full of said shills, since they have never even hinted towards a direct war against Russia, or wanted to get directly (by means of troops on the ground) involved in one.
Now I agree that sometimes it's simply not realistic to go all "well do it yourself then". But in this case, Ukraine accepts, trains and equips volunteers. It's not a hypothetical.
Amazing! We need more volunteers on the frontline: https://www.volunteeringukraine.com/en/volunteering-opportun...
[flagged]
Wasn't the large US/Canada blackout a decade or two ago caused by a tree?
If so, then the question would be if Russia did plant that tree. We should look out for more suspicious trees in our immediate areas.
See some tree squating where it shouldn't? Walnut or Vatnik? You can never be sure...
[flagged]
[flagged]
Tangentially related, my brain just shot off in that direction, the F35 must be a dream to target then, cyber-wise, for a competent state actor.
[flagged]
Threat from the state or another state? The uks biggest airport was shut down recently in a similar fashion after a fire at a power plant.
> For the past months the mass media has been talking non-stop about this possibility
Where? This is the first I have heard of it.
.. why would the state shoot itself in its own foot?
Local power outages are probably the most common "disaster" one should prepare for.
Source?
I can confirm, all coworkers from Portugal seem to be affected. Big thing.
Green energy! Maybe nuclear and fossil fuels would help?
[dead]
Local betting site had odds if this was a attack by russia
Europe still has many megawatts of solar with inverters that disconnect when detecting any grid disruption.
This is the absolute worst thing to do when there is a shortage of power - you immediately make the shortage worse and more grid disconnects.
The real fix is a grid with second by second pricing based on system frequency, and every individual user allowed to set a daily 'spend cap' of euros/dollars, letting them choose how much they are willing to pay for reliability.
Such an market has a huge stabilizing effect on demand, meaning a major incident would probably only have fairly small impacts on system frequency and embedded solar wouldn't disconnect.
You require inertia in the grid to maintain frequency and other stability stuff.
Solar PV is great but is mostly grid-following so cannot operate on it's own. As I understand it you need a minimum fraction of power generation to be large spinning turbines.
I think this problem can be mitigated with add-on rotational mass style kinetic energy batteries or something like that. I don't think variable energy pricing will help if it's an issue with over-demand the grid managers can do rolling blackouts to manage while fixing the supply problems. The grid is just broken at the moment and the solar can't maintain the grid alone.
https://en.wikipedia.org/wiki/Inverter-based_resource
Large Inverter Based Resources (IBR) such as huge solar parcs, grid-scale batteries or high-voltage direct-current (HVDC) lines can be programmed to behave like rotating generators, or even to smooth out smaller ripples. They also don't necessarily need a leading grid frequency but can be used to generate their own frequency normal to cold-start or resync a grid.
Only "small stuff" IBRs need a leading frequency from the grid and disconnect outside their safety corridor because those usually aren't controllable from some central grid authority. Thus the stupid-but-safe behaviour mandated for them.
While this is plausible, this solution works in the world of simulations.
We do not have a solid understanding of how inverter-based fast frequency response works with an existing grid that uses physics-based inertia.
What about large scale distributed containerised Iron Salt batteries?
I've seen some papers saying that they help stabilise grids.
This sounds more like a dystopian novel, where only the wealthy can afford stable power and everyone else is left without.
That is basically what happens in South Africa today. There are rolling blackouts and rich people with backup generators and batteries are unaffected. It's gotten to the point where richer cities (like Capetown) have their own power sources to mitigate the problem.
The current system is that everyone pays equally for X amount of stability, and for some people they don't get as much stability as they desire (eg. These people in Spain), whilst others pay more money and than they'd like when they would be fine with 5 minutes outages once a year.
That made sense before technology became available for everyone to make their own choice - but that is no longer the case.
Oh god.
Let's skip the technical problems in your theory and focus on the social.
People need power to survive. You know, food, hot water, light, work, internet, mobile phones, entertainment, etc. This requires stability, not second by second pricing.
When you put a chicken in an oven, you want to cook that chicken and eat it, feed your family. Electricity price rising in the next few minutes would mean that you either have to risk disease (chicken staying in the dangerous temperatures until the electricity price drops) or being hungry and throwing food away. This is not how you want society to function.
This comment is Dunning-Kruger effect in action…
Believe it or not, but maintaining an electricity grid is a massive undertaking, and the people in charge of it knows the topic much better than you do.
The problem isn't a market problem, it's a physics problem: having a synchronized grid of AC current with many producers over a wide area is a real challenge, even when the underlying issue is resolved it takes a lot of time to add the power plants (or renewable equivalent) to the grid because they must be synchronized.
The fact that the post you reply to includes such technical details as frequency-based pricing, indicates that the author has an above-average understanding of the technicalities of the power-grid.
Also, nobody in the field disagrees that in the more distributed grid we are seeing today, more endpoint communication and control could lead to more resilience. Whether pricing signals are the best path is a more open question, but they certainly appear to be a feasible option.
> The fact that the post you reply to includes such technical details as frequency-based pricing, indicates that the author has an above-average understanding of the technicalities of the power-grid.
No it doesn't. The fact that it's being said in a comment full of nonsense tells me that they don't have “above-average understanding”. They probably have read something, once, and now thinks they are an expert, that's literally what Dunning-Kruger is about.
They seem to believe that the equilibrium of supply and demand is all that matters, when it's just one piece of the puzzle and among the easiest to manage. Large, nation-scale, failures like this one are very unlikely to be caused by a lack of supply alone and markets are nowhere near fast enough to help preventing these.
Having such interdependent grids seems like a market or political problem not a physics problem to me.
The reason why we interconnect grids has nothing to do with politics and everything to do with physics, politics can sometimes lead to disconnections (like how the Baltic states disconnected from the Russian grid earlier this year) but it comes with great cost and involves careful planning (the fact that the Baltic states remained connected for almost three years after the start of the Russian invasion of Ukraine should give you an hint of how challenging it was).
The bigger the grid, the more efficient and resilient it is (and managing electric grids on islands is a nightmare), but it comes with a significant complexity and means restarting from zero is harder.
Interdependent grids are usually good: they allow you to average out the effect of a single power station failure over a much larger area, and to amortize prices from a wider area of suppliers.
Sure but if you can't cut off failing parts in a sane way it seems like a liability.
Like, what can you do, use some 1000 of MW to melt iron rods or something to give the power stations time to slow down? Free wheels?
What the heck are you talking about?
Don't you realize that the smaller the grid, the more important the instantaneous load variations can be in relative term and the harder it is to keep things running smoothly? It's not a theoretical concern, it's why electric networks on islands are much harder to work with and much more prone to collapse than bigger networks.
I think the point I am trying to make is that the risk for big area outtakes are lower with more figurative islands.
It is not. The bigger the network, the more stable it is, that's why countries keep interconnecting with each other despite political tensions between them (no pun intended).
Texas had a pretty free market, with many users on a floating price, didn't work that well either a few years ago when their grid went dawn.
EU electricity market is very similar to it, but interconnected and probably more free market style. A lot of people pay the spot price depending on county and their provider of course.
I didn’t really find anything problematic from an economic standpoint with the Texas power disaster. On the surface, the main problem there was just that a bunch of people were on spot energy plans, in order to save tremendous amount of money as long as the spot price stayed low. This is simply gambling. Those people after some period of savings suddenly cried foul when their gamble went south as prices spiked. The only issue I see here is if they weren’t adequately informed of that possibility. Obviously, the people who got the $2000 electricity bills are going to be the decide they weren’t informed adequately, and I’m sure it was spelled out in a document the size of an iTunes license agreement that literally no customer ever read. But anyway those people had themselves to blame as that plan was not mandatory or default.
This is the exact kind of techbro nonsense that I’ve come to expect from this website. No empathy for your fellow humans and dressed with a side of “computers and code plus finance”.
I thank the heavens that the people who run the electricity system do not share your opinions.
The "Blackout" novel comes to mind: https://en.wikipedia.org/wiki/Blackout_(Elsberg_novel)
Relevant:
https://www.pv-magazine.com/2025/04/22/spain-hits-first-week...
Apparently Europ network came close to shutdown due to frequency drop as the result: https://x.com/shellenberger/status/1916914966663434493?s=46
Except that's not true: an auto shutdown only occurs at 47.5 Hz, at least in Germany, some countries have the limit even lower: https://de.m.wikipedia.org/wiki/Unterfrequenz
Thanks for the clarification. However the idea behind this hypothesis is that the inertia stored in physical generators is mostly responsible for keeping the grid stable and as less inertia exists in the grid the risk of this goes up.
https://www.wsj.com/business/energy-oil/spain-portugal-power...
Sponsored by the fossil fuels industry... Also, Texans are not in a good position to lecture others about power network stability :-D
The official investigation is not over yet, but the head of the Spanish power network has already said that this cannot be related to renewables: https://www.lavanguardia.com/vida/20250430/10632617/beatriz-...
Is that a reliable source? The guy started blaming this on renewables hours ago even though no one seems to know exactly what happened.
WSJ good enough? https://www.wsj.com/business/energy-oil/spain-portugal-power...
Good enough for what? I've read the article. It's speculating that solar could be the culprit as it was the main source of energy at that time, but the grid operator couldn’t say what exactly failed, only that they had "two separate losses of power generation in southwestern Spain, a second and a half a part".
If you look at the generation stats[0][1], everything from nuclear to gas to solar dropped. And it was solar and wind that kept producing most of the energy for hours after the failure.
If renewables were the cause of this blackout, then fine, but that's not what the WSJ article says. The guy who wrote the tweet you linked to was blaming renewables 2 or 3 hours after it had happened and the tweet contained wrong information, as pointed out by a different comment here. To me, it's not a good source.
Just to be clear, if renewables were the main reason behind this blackout, then let's blame them and point out their weaknesses and see what we can do to mitigate them (batteries?). What we shouldn't do is jump to conclusions before we know what the heck happened.
---
[0] https://i.imgur.com/J375gbQ.png
[1] https://transparency.entsoe.eu/generation/r2/actualGeneratio...
I don’t think the argument (at least the one I’m curious about) is about renewables causing it but that with a growing portion of the grid coming from renewables a black start becomes more and more difficult.
I understand the physics behind this claim although I’d add a huge caveat that I’m not a power grid expert so put the comment here to see comments from experts
The guy is an American political journalist, his opinions here have exactly zero value to me.
His opinions or his facts?
His facts have no sources, so also worthless.
> Spain's electricity grid operator Red Electrica has confirmed power outages across the country.
That’s why you still need a strong diesel/diesel-electric locomotive fleet, imagine if Spain had been right in the middle military mobilization and military materiel transport, an event like this one would have stopped then dead in the tracks had they been relying only on electric locomotives.
No, the armed forces have their own fuel locomotives. I imagine any sizeable military does the same thing. Plus most (all?) cargo trains still use fuel. 100% electric is just passenger traffic.
https://www.outono.net/elentir/2021/10/23/the-spanish-army-r...
Maybe I'm looking at the wrong article, but there's this:
> For journeys outside the base, *the Army uses Renfe locomotives*.
which, in my understanding, means that in order to move military materiel (to the borders with France, let's say, or to the closest sea-ports most probably) and tens to hundreds of thousands of mobilised men the Spanish Army does indeed rely on Renfe locomotives, i.e not on their own.