I wonder how much smaller it could get with some compression. You could probably...

Rendello · 2025-12-30T22:27:56 1767133676

The hard-coded dictionary wouldn't be much stranger than Brotli's:

maxbond · 2025-12-31T05:45:14 1767159914

You can use a BPE variant like SentencePiece to identify these patterns rather than hard coding them.

jacquesm · 2025-12-30T22:23:11 1767133391

That's at least 45%, then you can leave out all of my comments and you're left with only 5!

hamburglar · 2025-12-31T02:31:36 1767148296

It might be a neat experiment to use ai to produce canonicalized paraphrasings of HN arguments so they could be compared directly and compress well.

keepamovin · 2026-01-01T23:22:14 1767309734

Dear it is already compressed using G zip – nine for every SQLlight shard and manifest

22 GB is uncompressed and compressed the entire things about 9 GB

rossant · 2025-12-31T06:45:03 1767163503

Guilty.