Hacker Newsnew | past | comments | ask | show | jobs | submit | porkloin's commentslogin

I've been a Kindle user since way back in 2010 with a Kindle 3G. I had a pretty huge amount of money invested into a library of books on Kindle. I installed koreader on my paperwhite earlier this year and haven't looked back. Even if it weren't for the limitations put on the walled garden for books in the kindle ecosystem, just the software upgrade alone feels great. When you hit buttons in koreader it actually responds in less than a second. I hadn't even realized how frustrating the UX of the kindle OS was until I realized the hardware was capable of feeling snappy.

There's obviously a lot you need to provide yourself that Amazon gives "for free" with the Kindle ecosystem, but using koreader with OPDS and kosync for synchronizing progress across devices, I'm very happy to be just buying DRM-free books elsewhere and hosting a small OPDS+kosync server on my home server. Installing tailscale on my kindle makes it so I can even fetch books/sync progress when I'm away from home without needing to expose my book server to the internet.


Quadlets are awesome and honestly I think one of the best additive things that podman has on top of the regular docker toolset.

I use podman regularly, and despite it being a good drop-in replacement like 95% of the time, the 5% of the time where it isn't seamless are super painful. For example, skaffold (https://skaffold.dev/) pukes all over itself when you try to run podman as a drop in replacement. I'm sure there are plenty of other examples, but that one stops me from using podman at work in addition to in my personal projects.


I have good results with this setup:

Hardware:

- GPU: AMD 7900xtx, 24gb vram

- CPU: AMD 5950x, AM4

- RAM: 64gb DDR4 3600

Software:

- OS: Bazzite (atomic fedora - this machine is running Steam "big picture" mode on my TV when not in use for LLM tasks)

- Virtualization: Podman Quadlets, which allows me to run container images as managed systemd units

- Network: tailscale

- Inference: llama.cpp vulkan (better performance than ROCM, though I'm keeping an eye on it in the future)

- LLM API surface: llama-swap (running as a podman quadlet exposed via tailscale svc) allows running multiple models on a single endpoint.

- Web/Chat Access: open-webui (running as podman quadlet exposed via tailscale svc) allows me to access any of the models I'm using for coding harness access for chat/general purpose queries via web browser. I also have the "conduit" app for my iPhone that allows me to hit the same models from my phone.

Models:

- Qwen3.6-27B-MTP-UD-Q4_K_XL.gguf - Unsloth Q4 quant of the qwen 3.6 27B model weights, with MTP enabled. MTP is important as it improves the speed the model can run at.

- Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf - Unsloth Q4 quant of 35B-A3B. Not MTP right now because I was having some issues with it?

- gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf - Gemma 4, which I use sometimes via open-webui instead of Qwen, but I generally think Qwen does a better job

Flags (specific for Qwen 27b, since that's primary model):

- `-ngl 99` offload all layers to GPU

- `-c 80000` 80K context window. I'd like this to be higher, but since my GPU also has to run the desktop session for the machine, I need to leave some VRAM overhead to keep the desktop from OOM-ing

- `-np 1` single slot (no parallel request handling)

- `--no-context-shift` error instead of silently sliding the context window when full

- `--cache-reuse 256` reuse cached prefix in chunks of 256 tokens (prompt cache)

- `-b 2048` logical batch size (tokens per submission)

- `-ub 1024` physical micro-batch (per GPU pass)

- `--cache-type-k q8_0 --cache-type-v q8_0` symmetric 8-bit K/V cache. Q8 is as low as I've been able to go without getting some issues with tool calling

- `-fa on` flash attention

- `--spec-type draft-mtp` use the model's built-in MTP as the draft model

- `--spec-draft-n-max 3` propose up to 3 draft tokens per step

- `--spec-draft-n-min 0` allow zero drafts if confidence is low

- `--spec-draft-type-k q8_0 --spec-draft-type-v q8_0` KV quant for the draft path

- `--reasoning-format deepseek` parse <think> blocks in proper format

- `--chat-template-kwargs '{"enable_thinking": true}'` turns on Qwen's thinking mode on by default (clients can override)

- `--jinja` use the GGUF's Jinja chat template

- `--temp 0.6` moderate randomness (Qwen recommended value for coding)

- `--top-p 0.95` nucleus sampling (Qwen recommended value for coding)

- `--top-k 20` top-20 candidates (Qwen recommended value for coding)

- `--min-p 0.0 disabled (Qwen recommended value for coding)

Performance (27b, primary model):

- ~65t/s for token generation

- ~600 t/s for prompt processing.

- If these numbers don't mean much to you, perceptually this feels about on-par with cloud model speed, maybe slightly faster.

- ~30s cold start when swapping from a different model or starting up session from idle via llama-swap.

I have llama-swap set up to unload the model after 10 min of idle, because I sometimes use this machine for gaming as well. A little annoying, but a small price to pay to be able to use the machine for other stuff (gaming) when I'm not using it with coding tasks.

CLI/Harness:

- Crush harness (https://github.com/charmbracelet/crush) less feature rich than Claude Code, but with a smaller system prompt and better built-in LSP support. I point it at the tailnet DNS (https://llama.<tailnet>:<port>)

- Headroom (https://github.com/chopratejas/headroom) to maximize the 80k context window

- Exa MCP for web search (https://exa.ai/) this alone makes the model far more useable. It's shocking how often the official claude code or codex harness get botblocked on web fetches, and the results of a good web fetch can be the difference between a good turn and a bad turn.

A lot of people get hung up on whether Qwen 3.x models are "as smart as" some parallel Anthropic model. Most people seem to agree it's somewhere between Haiku 4.5 and Sonnet 4.5. Personally, I think the biggest thing that makes the Qwen 3.x series of models _feel_ good to use for coding workflows is that its the first time that tool calling actually works consistently on local models. If tool calling is busted even 5% of the time, it can totally ruin the flow. I think that's also why people tend to say the "harness is more important than the model" or whatever. I have a few other models set up but 27B with MTP is the best compromise of speed and quality that I've found.

This setup works well enough for me that I dropped my personal Claude Code subscription. At work I'm still using frontier models, but personally I don't feel like I need that much power for anything I work on in my personal life. I'm "lucky" that I made the random financially unwise choice to buy a 7900XTX in late 2022 for $1k as a gaming card. I had no clue it would actually be a pretty decent LLM card 3-4 years later.

Edit: sorry for the horrible formatting, I always forget that HN doesn't actually do markdown :(


Now that's what I'm talking about! Very cool, thank you for the detailed response.


The idea that Japan is a uniquely "homogeneous culture" is honestly a modern construct anyway. Japanese culture and language has been enormously influenced by colonial and migrant presence in the country, from Chinese to Dutch to British to American, and a zillion others.

Just look at the language! I don't have the exact figure in front of me, but I remember when taking Japanese language courses that something like 30% of the lexicon is loanwords from other languages (edit: I looked it up and it's apparently closer to 50%) Way higher than most other widely spoken languages on the planet. Japanese culture is legitimately _amazing_ in its capacity to absorb and domesticate outside influence, and it's unfortunate that people both in the country and abroad are so short-sighted to not see that.

The Meiji and Showa era militarism benefited a lot by promoting this myth. They weren't alone, mind you. Lots of folks across the EU and the US are still falling for the same nationalist stories that their governments cooked up in the early 1900s to drive them all to war.

The country _does_ have a really notable cohesion and shared identity, but the problem is in attributing that to some kind of unique isolationism rather than their long history of pluralism.


I think you're misunderstanding - "homogeneous" as a cultural description is a spatial metric, not a time metric. Homogeneous cultures indeed change across time, influenced by war and foreign media/languages, just like any culture. The point is not that they do or do not change, the point is that they do so together, largely.

>The country _does_ have a really notable cohesion and shared identity

This, and race/culture, is what is meant when people say Japan is homogeneous compared to other developed nations.


I think you are confusing "homogeneous" with "indigenous". With the common meaning of "homogeneous" your entire post is a non sequitur. There are many different words ending with "-geneous" meaning entirely different things, you need to look at what goes before the "-genous" part.


>The idea that Japan is a uniquely "homogeneous culture" is honestly a modern construct anyway.

This is a very postmodern point view.


I was going to say, Japans history is one that was very... combative. They fought amongst themselves heavily for a long time.


I think a lot of places were at various points in time, right? It's always a little easier to paint with a broad brush and lump hundreds of years into a single statement when you're talking about the history of "foreign" places that we don't learn the history of very deeply in western education. For Japan in particular, it's hard because a big part of the Meiji nationalist movements was to recast Japanese history with a heavy focus on the "bushido" and an arguably manufactured version of some points in the country's history that _were_ undoutably bloody. That yarn-spinning from 100+ years ago has actively shaped how the rest of the world thinks about Japanese history. Western governments during WW2 were happy to take that narrative and paint the entire history as blood-soaked and brutal to dehumanize their enemies. But it's not hard to find evidence of how many long stretches of internal peace existed in Japan. After the establishment of the Tokugawa shogunate there was 250 years of more or less continuous peace internally. Arguably continental Europe from the middle-ages onward is more fractured and bloody in total than Japan was in the same period. Classical Greece was a zillion times more bloody.


assuming you have a locally running llama-server or llama-swap, just drop this into your crush.json with your setup details/local addresses etc:

Edit: i forgot HN doesn't do code fences. See https://pastebin.com/2rQg0r2L

Obviously the context window settings are going to depend on what you've got set on the llama-server/llama-swap side. Multiple models on the same server like I have in the config snippet above is mostly only relevant if you're using llama-swap.

TL;DR is you need to set up a provider for your local LLM server, then set at least one model on that server, then set the large and small models that crush actually uses to respond to prompts to use that provider/model combo. Pretty straightforward but agree that their docs could be better for local LLM setups in particular.

For me, I've got llama-swap running and set up on my tailnet as a [tailscale service](https://tailscale.com/docs/features/tailscale-services) so I'm able to use my local LLMs anywhere I would use a cloud-hosted one, and I just set the provider baseurl in crush.json to my tailscale service URL and it works great.


I think Kagi is kind of making this happen currently with search. Not sure how their adoption number are going, but people are willing to pay $$ for better search with no "sponsored content" rising to the top.

I'm hesitant about a lot of this stuff because it's very easy to get to a place where we let net neutrality degrade even more than it already has. Part of the way that platforms indoctrinate us to accept that paying extra for quality of service or "fast lanes" for specific content types are "necessary" is to degrade the existing experience so much that it seems inevitable.


Good catch. I didn't even think about the fast lanes fiasco. I don't know why businesses have decided that since they have connected to the internet that the internet owes them.

It should be a public utility. It should be as ad free as reasonable. It should not track you.

The internet should be a lot of things that it currently isn't all because rent-seeking money and power grubbing bastards have too many of the strings and love pulling them like they're pulling their puds.


I swear by Kagi and will never go back (until they inevitably start including ads after a bad earnings report)


Honestly even in "developed countries" it's not worth blindly trusting that the power in your house/building is clean. It's cheap and easy enough to just put any expensive hardware on a UPS rather than speculating what's going on behind the walls.


I work on embedded systems. I can often see whether my A/C or other appliances are running on my oscilloscope signals. They often affect the output of USB power supplies.


Eh, if these surges are rare enough, then you are statistical better off just risking your 'expensive' hardware to a one in a trillion possibility rather than spending money on gear you don't need.

Do you live in a bunker to protect against artillery shells?


Doesn't even sound like a developed country to me. Is that the US or something?


I think it's important because there are a bunch of would-be claimants for intellectual property violation. Many people speculate that their work was used in training data, but it can be difficult to produce sufficient proof that their copyrighted work is present in the training data. If you could reliably get an LLM to produce 70% of a copyrighted book that would probably be enough to get a few lawyers salivating.

I didn't read the source paper referenced in the ars technica piece, but this statement about it makes me wonder how useful it actually is:

> But a study published last month showed that researchers at Stanford and Yale Universities were able to strategically prompt LLMs from OpenAI, Google, Anthropic, and xAI to generate thousands of words from 13 books, including A Game of Thrones, The Hunger Games, and The Hobbit.

It seems like well-known books with tons of summary, adaptations into film scripts, and tons of writing about the book in the overall corpus make it way less surprising to see be partially reproducible.

So I guess that's a lot of words to say - yeah until there's something definitive that allows people to prompt LLMs into either unlawfully recreating an entire work verbatim or otherwise indisputably proving that a copyrighted work was used in training data, there's probably nothing game changing in it.


It's well-known books, yes, and even then with significant errors which means presumably lawyers for the AI companies would argue there is no possible damage. That said, US copright law has statutory damages for registered works that are not based on real, documented damages. I could totally see it being fought over, but I also agree it's probably not going to end up being game changing.

I suspect very works will be memorised enough to be an issue, and we'll see the providers tighten up their guardrails a bit for works that are well known enough to actually be a potential issue (issue in the form of lawsuits, not in the form of real damages to the copyright holders)


Clearly discord has more of a vested interest in boosting engagement - especially now that they are showing people "quests". What a quirky and fun way to say "ads"!

But at the same time I don't necessarily buy the idea that all of their reactions/roles/badges/etc are exclusively malevolent engagement-driven design decisions meant to hook people. I do think that some of them are legitimate improvements to chat communication, and as a result many of those features have proliferated across other messaging platforms. Hell, most of them didn't even originate at discord at all but were cribbed from their competitors.

To be clear, I 1000% agree with you that IRC is less addicting. Even just by simple merit of not having multi-device push notifications. Those pull me back into the app. But push notifications across devices are also just objectively useful. I name that one in particular because it's one of the biggest and most notable features that prevents me from returning to IRC, where I happily did most of my chat until the mid 2010s. I'm actively shopping for a discord alternative as a regular user who is fed up with discord's march toward enshittification, and matrix looks like it gives me most of that convenience without the worst parts of discord.


The guy is high on his own supply. This entire thing reads like a fever dream.


Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: