I'm very curious about the convergence of the best in class fast OLAP products (StarRocks, ClickHouse) with Trino. It sounds like everybody is going for decoupled compute/storage, using S3 or similar as the storage layer, and thus forgoing colocated joins (ok I know ClickHouse joins suck)...

So what does this mean for ingestion (and indexing)? Iceberg V3? Paimon? Bespoke ingestion through the DB engine to do the indexing?

It's also interesting how Clickhouse / Starrocks can now also act as a query planner and executor on top of non-native formats (ex. Iceberg).

I assume the native formats will always be faster / more optimized but the need for Trino as a separate executor while running either of these databases seems to be close to gone.

The benefit of StarRocks/ClickHouse over Trino is that you get secondary indices, but that means you have to do the indexing somehow.

Native format is faster (especially for colocated joins), but it's way more expensive if you have to run a bunch of separate storage nodes vs just using S3, especially your query volume isn't that high.

I liken it to the BigQuery cost model, where storage is effectively free.

> I assume the native formats will always be faster / more optimized

Actually query on Iceberg is end-to-end faster on ClickHouse than MergeTree native format on disk. It's mostly a matter of how much compute you throw at it. Native MergeTree still wins in cases that depend on use of indexes to reduce I/O but scan speed is no longer an issue.

I’ve historically read this as ‘open format compatible’ but ‘native preferred’ - where this opens up market space and dev velocity - but it’ll be interesting to see if native storage differentiation gets dumped entirely. It just seems like ‘fork and optimize for our engine’ would always be tempting enough that you’d want a native play for when you don’t need the decoupling.

Clickhouse joins have been improving almost every month for the last couple of years. Maybe they still suck but a lot less

Apparently engineers are running out of tokens with our mediocre token budget, and there is talk of trying to run local LLMs.

Meanwhile execs get unlimited tokens...

And here I am, only automating the most mundane and boring stuff so I can program by hand, which I enjoy.

You are absolutely right to push back...

They used to care about developers. Peak cocoa and uikit was a thing of beauty and objc remains my favorite programming language of all time. When Swift came along i tried and was dissapointed and then a decade later i gave it another whirl and was shocked it was still so bad. The hardware has gotten better, but the software clearly not.

Country of lawers vs country of engineers

Nah, gonna be populist idiots from the other side.

The same way we dont want smart people on juries, smart people in government is haram. especially people with technical educations who care about privacy. very haram!

You forgot the police with its explicit maximum IQ limits.

This is not a real thing, but people on the Internet badly want it to be.

You mean token holders

Any reason SpaceX cannot just build a giant solar shade at L1? This seems fairly doable on the decade timescale, plus it could do double duty as a weapon.

I did the math out on this for fun a couple years ago. It came put surprisingly reasonable.

Taking estimates for Starship payload to Mars as the likely payload to L1, carrying to L1 enough thin reflective mylar film to block around 0.5% of light to earth would cost between $80-100 billion. Which is a lot but not nearly as much as I expected and not outside the realm of possibility.

I love how you didn't even take a beat before suggesting that it be used as a weapon.

"in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying Beware of the Leopard"