The default kubelet `syncFrequency` is 1 minute (per Pod). (There can be additional event-driven ones, but this is a floor.) Back of the envelope: 30 Pods * 1440 * 7 = 300k calls to startPodSync in a week.
I’d guess a lot of production clusters don’t run the latest release of Kubernetes, and 1.36 was only released in late April, less than two months before I reported the issue.
Given that the issue shows up as linear memory growth on the order of maybe 1 GiB/month per node, there really is a time-based discovery process here. (Could be artificially accelerated for a CI test, I imagine!) And any restarted nodes or clusters that upgraded point releases would restart that clock.
The only unusual contributing cause that made the issue more visible to me was trying to run everything on a tiny 2 GiB RAM node!
At the metrics level, you can compare old vs new release. Have been bitten before by resource requirements dramatically change (regardless of whether it's a bug or functionality change)
Thanks. I know there's a `go vet` tool that's run as part of Kubernetes CI, and one of its checks is:
lostcancel: check cancel func returned by context.WithCancel is called
I'm not 100% sure why `go vet` didn't catch this issue, but storing the cancelFn in the struct is probably part of the reason. Any Go experts know if that's the case?
The cancel function escapes the function body, so static analysis can't detect it. There's another lint for that (containedctx), but I think it's off in K8s.
This is a serious tripping point with Go. There's no way to express: "this is a root context that I _want_ to store and only use to create derived contexts". Goroutines are also a source of problems, you can't easily say "I'm passing the ownership of this context to a goroutine".
I took a quick look at "containedctx" and it seems like for this case, it would almost be backwards: it would flag the (not-memory-leaking) struct-stored "status.ctx", but wouldn't flag when there is a stored "status.cancelFn" only (which resulted in the memory leak).
This actually is possible now. Contexts are now garbage-collectible, even if cancel() is not called.
In this case, the cancel() function was preventing the collection. But I think it can be changed to hold a weak reference instead. The overhead is too large to run it normally, but it should be OK for something like the race detector.
Author here! If you're running a Kubernetes cluster, I recommend you check `kubectl version` and see if you're running "Server Version: v1.36.[0,1,2]". If so, you may want to use the one-liner at the end of the article to check your "process_resident_memory_bytes" on each node, and consider restarting kubelet as a temporary workaround to tame the memory leak until v1.36.3 is released.
Here are some N-year rolling total inflation charts to put this datapoint in a longer-term perspective: https://totalrealreturns.com/inflation . Zooming out always smooths the noise.
> Prices are up +4.25% in the past year, and +24.49% in the past 5 years, according to the latest CPI data released Jun. 10, 2026. The price level has approximately doubled (2.01x) today compared to August 1999.
Not knowing if that's good/bad, as it is without any frame of reference, so the same data for Spain looks something like this:
Prices up +3.2% in the past year, up +22.4% in the past 5 years. Compared to 1999, a 1.88× difference, and if you want to compare since when it doubled, it'd be around September 1996. This is according to a tool from INE, Spain’s national statistics: https://www.ine.es/varipc/index.do?L=1
It depends on the country. Brazil, due to its hyperinflation days, has a lot of indexed prices. These are prices that automatically increase due to inflation. This makes the country to have so called inertial inflation, current inflation caused by past inflation, and also makes it more robust to a higher inflation.
Can you really say that based only on the inflation? What if wages increased 6%, then 3% inflation wouldn't be as bad as if inflation raised 2% but wages only increased 0.1%? At least if you think about purchasing power I suppose. But won't claim to be an expert on this, happy to be educated by those who are :)
Part of it is just expectations btw. If the value of money is jumping up and down rapidly it is bad for business. Like if I'm going to sell you a 30 year fixed rate mortgage we require an accurate expectation of future inflation for one of us not to effectively lose their shirt on the deal. Imagine shops shuffling their prices up and down constantly, unions renegotiating contracts all the time, you sign up for 12 months of netflix but the price implicitly assumes that money will be worth N% less by month 12, etc. (imo a lot of these things should already be pegged, but people don't like doing that) It's basically just much more annoying to transact using a currency whose future value is unpredictable. So given that 2% is the stated target, which expectations are presumably largely built around, significant deviation is a failure to manage that process.
In general, higher inflation has a negative impact on consumer sentiment even if wage growth matches the inflation, which it rarely does.
But the bigger issue is that inflation is generally distributed much more evenly than wage increases. Very few employers offer a COLA that is automatic, so wages almost always trail inflationary pressure.
It's fine as long as t-bill rates match or exceed inflation. Then you can avoid losing purchasing power by just putting your money in the world's safest investment. Over the past century, t-bill returns have slightly exceeded inflation on average, though there have been periods when they didn't.
Stash paper cash in your safe and sure, you lose purchasing power. Use fiat money the way it's designed to be used, instead of using it like gold coins, and it works better.
Us debt as a fraction of GDP has doubled this century and roughly quadrupled in my lifetime. It would seem to me that eventually t-bills will not be safe.
If currency doesn't devalue then stuffing it under a mattress looks like a reasonable alternative to investing. If we hit deflation you can receive gains for "free" and borrowed money becomes more expensive over time. Neither of which our economic system is setup to handle.
We punish people who hoard cash by devaluing it thus encouraging them to put the money to work.
One side says this design is necessary to sustain growth. The other says it's unfair because the gains from the growth are unevenly distributed. Neither is wrong.
Why is that not good? When inflation is close to 0 real interest rates increase which causes the economy to slow down. It seems clear to me that the optimal rate of inflation is always above 0.
The real problem imo is that below 0% is really bad, and has the potential to spiral. So the fed does not target anything close to 0%, but instead targets some buffer above it.
So it's not that "2% is good", but more that "2% is the best buffer we've decided above the <0% super scary threshold"
Yes of course below 0% is especially bad, but I dont think thats the whole story. If central banks were able to set inflation with 100% certainty I still think targeting a number close to 0% is a bad idea. Nominal interest rates have a floor due to defaults, servicing costs. As inflation approaches 0 that floor is hit and monetary policy loses its ability to control real interest rates. Keeping nominal rates above their floor is key to ensuring small business can obtain liquidity, as the floor is approached it makes less sense for lenders to write small loans.
There are many other reasons a positive inflation rate is better than substantially near 0. One common complaint about inflation is that erodes real wages because nominal wages are sticky, but this is actually a good thing. It gives businesses room to breathe during downturns without cutting nominal wages or having to cut staff. Positive inflation also forces cash into productive uses which helps monetary policy because it keeps the actaul money supply more stable.
The Fed did a study some time back estimating CPI levels since 1800. [1] They found that from 1800 to 1950 the CPI never shifted more than 25 points from the starting base of 51, so it always stayed within +/- ~50% of that baseline. That's through the Civil War, both World Wars, Spanish Flu, and much more. And obviously the US economy increased in sized quite exponentially from 1800 to 1950, with no persistent inflation whatsoever.
It's even more interesting to contrast this from 1971 onward. 1971 is when Bretton Woods ended and the government was given a free hand to start 'printing money' so to speak, and inflation became the new policy. Since then the CPI has increased by more than 800 points, 1600% more than our baseline. And it's only increasing faster now - to the point that these numbers I'm giving are already rather outdated.
The US economy faced repeated economic catastrophes from 1800-1950 largely because the government was unable to enact monetary policy. The long depression of the 1870s happened pretty much solely because monetary supply contracted and populists got elected to fuck with the silver/gold standard. Causes of the great depression are more varied, but contraction of money supply due is certainly one of the leading ones.
Yes, the economy expanded greatly over this period, but you have to separate inflation from many other causes such as innovation, increasing labor supply, better education, increases in the amount of investment. I think its pretty clear that the economy wouldve fared much better in the 1800-1950 period if the government was partaking in monetary policy that focused on small but positive inflation.
Check out the data from the Fed and contrast it against events in the past. For instance you mention the long depression which happened from 1873 to 1879 and resulted in a decline in prices of about 30% followed by stabilization. And of course that was also the advent of the Gilded Age, where economic growth, wages, and so on all were skyrocketing, all while prices trended downward! It's difficult to even imagine something like that now a days.
I don't think that's just a coincidence either. Economic issues in the US used to foreshadow booms to come, which makes sense in many ways as it's the ending of one generation of businesses and the start of another. By contrast in the US prices have increased by 30% over almost the same length of time as the 'long depression', and continue going up up and away. It'd be nonsensical to call it the long inflation or whatever because it's only slightly off the normal. 2% 'planned' inflation over the same 7 year period is a 15% increase in prices. And businesses going under? No, everything's huge now, and so everything's too big to fail. The government has taken on the responsibility of perpetually propping up failing businesses, forever inhibiting competition in the process.
And there's no boom waiting at the end, in no small part because there is no end. Each economic issue we face, which are becoming increasingly regular, just further magnifies the divides in society. The wealthy have sufficient assets and resources to turn this into profit, in no small part by dumping excesses of money into inflation resistant assets, but the middle and lower classes have no such option and so mostly just lose either badly or very badly. This [1] site lays out a bunch of the data since 1971 (when the dollar became completely unbacked following the end of Bretton Woods) and impossible not to see it as an enormous inflection point for all sorts of nasty stuff.
---
I suspect if we hadn't had the tech boom kicking off fairly shortly after 1971, driving in a decades long unprecedented economic boom, that this experiment would have long since reached its climatic failure. It only works with infinite exponential growth. For some time we had that. Now we not only no longer do, but also are seeing a fertility collapse at the same time. There's gonna be some fireworks.
The broad idea is you want a number low enough that people don't price inflation expectations into day-to-day pricing but not so low that a hiccup causes deflation.
The empirical evidence around inflation persistence is a bit all over the place, but broadly suggests people start daily indexing between 2 and 5%. When that starts to happen, restraining inflation without causing a depression becomes incredibly hard, because people will actively countermand policy moves.
The fed has a dual mandate to maintain full employment and keep inflation at 2%. Others have already explained why 2% and not 0%. Up to 3% is expected, 4% means significant price shocks and they should consider acting quickly. 5% means they are at risk of losing control of inflation as it's more than doubled from their mandate and the fed risks losing credibility with markets
Inflation isnt as simple as good/bad. Monetary theory shows us that short term inflation is a good way to counteract spikes in unemployment. Whether you prefer stable inflation with swings in unemployment or stable unemployment with swings in inflation or something in between is a political question.
That shows that it’s been since 1991 since we saw similar five year increases in prices. Which is a long time. You also have to be careful not to zoom out so far you get into the “we all die anyway” scale where you’re not really tracking things that are meaningful to on-the-ground, as-lived reality
1918 isn't very relevant to modern living. And nobody wants to go back to the stagflation of the 1970s. And that scale is logarithmic.
Graph it without the logarithmic scale and draw a curve through the 1982-2018 data and the recent spike will explain why people are complaining about it.
Indeed. Back then food and shelter comprised a much larger % of the average income, and so each percentage point of inflation was considerably more painful than it is now.
Zooming out in what sense? Those rolling charts don’t mean much imo. Year over year change is a pretty good perspective and a tick up like this is not great.
Zooming out tells you this tick up to 4.2% is not nearly as bad as the post-covid inflation, and drastically better than the 70s. Not a good sign, but also not too far outside the historical mean and probably no need to panic in and of itself.
Obviously more inflation is bad almost always. The question is how bad. By looking at a graph, you can get a sense for the mean and standard deviation of inflation over time, and this looks to be a small deviation in the grand scheme of things. Of course, no one would call this good, but it's just good to put stuff like this in context.
Again I don’t think it says anything. Yes inflation was worse during Covid and before I was born, that just sounds silly btw. There is no perspective to be frank, energy is up a lot, that will impact other goods over the next few quarters.
I am not sure what the perspective is: we aren't the same economy (there are true financial system differences between now and say, 1985) and, even if we were the same, the three other shocks that rise like this are two world wars and an oil crisis. This is some dunderding old narcissist thinking he's the toughest kid on the block. You could argue the oil crisis was a similar result of the US never, ever learning a lesson about intervening in others' political systems (especially if there's oil involved), but trend line or not, no one had to go through this.
And the trend line would bend differently if we could just learn the lesson.
And yes I am oversimplifying: the current conditions are actually do to a number of stupid things the current administration did because they assumed everyone who came before them was stupid and woke, but this just strikes me wrong, as though the chart should be comfort to someone struggling to make rent or pay for medicine or what have you. Much of this could have been avoided.
> Redis (and memcache) are memory caches and should be treated like that
If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrockshttps://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space.
As I've been improving my side project https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.
Re: "page for all 500s": there's a world of difference between "page me with a critical alert at 3am" and "notify me on Monday morning when my normal workday starts". At the extremes:
If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am!
If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monday.
Not always easy to distinguish those clearly or configure those business hours rules, but for my team at https://heyoncall.com/ that is the goal -- otherwise your team burns out fast. Waking up someone at 3am has a real cost, so you better be sure it's worth it.
One team can't troubleshoot AND FIX every possible subsystem, so you just end up with lots (growing to hundreds) of people "on-call" anyway.
As others have said, follow-the-sun type models do exist, usually staffed by people in their normal working hours (EMEA, Americas, APAC) but this means you've still got to cover the weekend and public holidays (which there are a lot of when you factor in plenty of different countries).
Where you need a quick response you can have a core ops/noc team that looks at things with lower thresholds and shorter windows, and their job is to do the initial triage and then page the appropriate team earlier than they would have been alerted by their own alert thresholds/monitoring.
Actually clicking the button to change the status on a public status page is a whole different topic that becomes very political in certain companies.
I've worked in large orgs where we could (at at some times did) have around the world rotations. They don't work well. It've very hard to maintain real team cohesion, and you end up with really superficial operations. People tend not to dig in really deep, find good fixes, etc. Lots of superficial bandages.
You can cheaply and readably give a lot of clues to both agents and humans with some assertions at the start of a method:
raise ArgumentError.new("...") unless ...
which can include type assertions but also a lot more. The agents seem to do well with this.
I've also had good results using agents to write Crystal https://crystal-lang.org/ which is Ruby-like but does have the static types and produces blazing fast static binaries. Might be a sweet spot for coding agents if you're building some backend services. But I'd still pick Ruby on Rails for a new full stack project.
The edges drawn are between nodes that have been able to hear each other in the last 24 hours, based on observed traceroute packets.
(Even then, it’s only a subset of the actually-connected nodes: the map only shows nodes that have published their position on the public channel, and have set a flag that their data is okay to uplink to a server over MQTT.)
It can be dangerous to backfeed (which is why you're supposed to have an interlock for a generator inlet, ensuring utility power is disconnected). But:
1. These grid tie inverters are designed and tested to shut off completely if there's no grid power. (This is a big design tradeoff: it means they don't provide any power during a grid power outage, even if it's very sunny out.)
2. Even if I had a beefy generator that was unsafely backfeeding my house while the utility power was still connected, the generator would be trying to power not just my house, but all my neighbors too! And the circuit breaker and/or inverter on the generator would likely trip and shut down almost instantly.
There's still a possible risk from #2, especially if the downed wire being repaired is relatively local (i.e. your house only).
But I think #1 and #2 mitigate this risk very well.
On point 1, you can pay extra to get an inverter that does "islanding mode". During an external outage the inverter stops sending power out of the house but keeps supplying power inside the house. Whole-house backup batteries such as the Tesla Powerwall (and competitors) also have this capability.
Anyone who is looking for a generator to power their house during a regional outage should look into other types of generation that will do islanding mode.
The default kubelet `syncFrequency` is 1 minute (per Pod). (There can be additional event-driven ones, but this is a floor.) Back of the envelope: 30 Pods * 1440 * 7 = 300k calls to startPodSync in a week.
I’d guess a lot of production clusters don’t run the latest release of Kubernetes, and 1.36 was only released in late April, less than two months before I reported the issue.
Given that the issue shows up as linear memory growth on the order of maybe 1 GiB/month per node, there really is a time-based discovery process here. (Could be artificially accelerated for a CI test, I imagine!) And any restarted nodes or clusters that upgraded point releases would restart that clock.
The only unusual contributing cause that made the issue more visible to me was trying to run everything on a tiny 2 GiB RAM node!