great question and thanks for checking it out! I've talked to a number of folks who have built small/simple versions of this for various workflows.
The idea for incorporating feedback into the knowledge base is still coming together, in the prototype, LLM can classify the response as approval or not, and then if it's a rejection, llm will try to distill out facts/ideas from the response, e.g. "BigCorp and Acme.com are also using XYZ product" or "to learn more about pricing, you can book a meeting at LINK".
In the prototype, it then did a function call to add those as small chucks to the vector store, but you could also orchestrate that transparently if you didn't want to rely on the LLM reliably calling an `add_to_knowledge_base` function.
Longer term, I like the idea that I first heard of in BabyAGI, which is to store the messages leading up to an approval + the approval result in a vector DB, and use those historical approvals to derive up a confidence score for whether a particular action will be approved.
That stuff's more whiteboard stage than in code yet but I think it could be built.
The idea for incorporating feedback into the knowledge base is still coming together, in the prototype, LLM can classify the response as approval or not, and then if it's a rejection, llm will try to distill out facts/ideas from the response, e.g. "BigCorp and Acme.com are also using XYZ product" or "to learn more about pricing, you can book a meeting at LINK".
In the prototype, it then did a function call to add those as small chucks to the vector store, but you could also orchestrate that transparently if you didn't want to rely on the LLM reliably calling an `add_to_knowledge_base` function.
Longer term, I like the idea that I first heard of in BabyAGI, which is to store the messages leading up to an approval + the approval result in a vector DB, and use those historical approvals to derive up a confidence score for whether a particular action will be approved.
That stuff's more whiteboard stage than in code yet but I think it could be built.