Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
Uehreka
66 days ago
|
parent
|
context
|
favorite
| on:
AI will make formal verification go mainstream
No, it’s an example that shows that LLMs still use a tokenizer, which is not an impediment for almost any task (even many where you would expect it to be, like searching a codebase for variants of a variable name in different cases).
8note
66 days ago
[–]
the question remains: is the tokenizer going to be a fundamental limit to my task? how do i know ahead of time?
worldsayshi
66 days ago
|
parent
[–]
Would it limit a person getting your instructions in Chinese? Tokenisation pretty much means that the LLM is reading symbols instead of phonemes.
This makes me wonder if LLMs works better in Chinese.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: