We've got to stop calling this a "Google Memo." That's a false narrative. It's just a random doc written by one of 140000+ employees.
> Google has been contacted for comment but it is understood that the document is not an official company memo. [1]
There are moats to products, but less so to pure language models trained on the same web-scale scraped data that many share.
Not all data is readily available to language models, and integration can be difficult.
A company that specializes in say AI for trash sorting likely still has a moat.
Microsoft integrating AI into Windows still has a moat (for Windows).
GPT-4 is ~200 Elo better than the next best semi-public Vicuna-13B in Chatbot Arena [2]. That is a non-zero moat - perhaps due to hosting larger models, training data, licensing, output postprocessing, etc.
> GPT-4 is ~200 Elo better than the next best semi-public Vicuna-13B in Chatbot Arena [2]. That is a non-zero moat
Its a non-zero advantage.
A moat is something that inhibits someone from closing an advantage.
(Also, its odd that the biggest models, outside of the big vendor centralized ones, they are testing are 13B-14B when 30B-ish and 65B-ish versions exist.)
However, if the advantage is due to things like inference infrastructure to support a massive model, that isn't easy to duplicate.
I would also say that the quality of these smaller models are good, but we also may not be measuring them correctly. Recent papers suggest that these smaller LMs dont fully capture ChatGPT quality in ways that may not have appeared with crowd worker ratings [1]. It's easy to have your inputs be inside a happy distribution for a paper but fail in the real world in ways that GPT-4 doesnt.
Lmsys would love to compare with bigger models but have limited resources. Contributions are welcome [2]
The moat (or the water/crocodiles in it) is the content that the company gathers in relation to the offering that is being defended. Microsoft has Github, which is a source of code that the model can operate upon, as well as the interactions/queries with the users. OpenAI is playing around with sharing content because of this. They want to build a moat and will use us to do it.
If someone just has an approach to solving the problem, i.e. code that does this that and the other, then there is no moat.
It is a memo written by a Googler on internal systems that makes it a Google memo. Companies almost never officially sponsor internal memos escaping without PR and legal having a crack at the content.
What I'm really curious about is why you think this isn't a Google Memo, and why you think that's a false narrative.
There is a huge difference between a leadership-endorsed strategy memo like the Nokia/Elop "Burning Platform" memo (1), and a memo by a random engineer like Steve Yegge's Platform rant (2).
Random engineer memos can certainly still be influential, but they do not dictate company direction.
To me, "memo" implies it being for the business. If this was written by leadership or intended to be some sort of instruction to others, then I think it would be more reasonable to call it that.
But it wasn't. It was an opinion written down by an individual, for no other reason than to share their personal opinion. It was closer to an HN comment, not a document necessary as part of business operations.
Anywhere I've worked, junior engineers have put together ridiculous memorandums, especially when they're on the way out. That does not mean that the opinions of the author align with the company's.
> Google has been contacted for comment but it is understood that the document is not an official company memo. [1]
There are moats to products, but less so to pure language models trained on the same web-scale scraped data that many share.
Not all data is readily available to language models, and integration can be difficult.
A company that specializes in say AI for trash sorting likely still has a moat.
Microsoft integrating AI into Windows still has a moat (for Windows).
GPT-4 is ~200 Elo better than the next best semi-public Vicuna-13B in Chatbot Arena [2]. That is a non-zero moat - perhaps due to hosting larger models, training data, licensing, output postprocessing, etc.
[1] https://www.theguardian.com/technology/2023/may/05/google-en...
[2] https://lmsys.org/blog/2023-05-25-leaderboard/