Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

No, it’s an example that shows that LLMs still use a tokenizer, which is not an impediment for almost any task (even many where you would expect it to be, like searching a codebase for variants of a variable name in different cases).


the question remains: is the tokenizer going to be a fundamental limit to my task? how do i know ahead of time?


Would it limit a person getting your instructions in Chinese? Tokenisation pretty much means that the LLM is reading symbols instead of phonemes.

This makes me wonder if LLMs works better in Chinese.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: