great question and thanks for checking it out! I've talked to a number of folks who have built small/simple versions of this for various workflows.
The idea for incorporating feedback into the knowledge base is still coming together, in the prototype, LLM can classify the response as approval or not, and then if it's a rejection, llm will try to distill out facts/ideas from the response, e.g. "BigCorp and Acme.com are also using XYZ product" or "to learn more about pricing, you can book a meeting at LINK".
In the prototype, it then did a function call to add those as small chucks to the vector store, but you could also orchestrate that transparently if you didn't want to rely on the LLM reliably calling an `add_to_knowledge_base` function.
Longer term, I like the idea that I first heard of in BabyAGI, which is to store the messages leading up to an approval + the approval result in a vector DB, and use those historical approvals to derive up a confidence score for whether a particular action will be approved.
That stuff's more whiteboard stage than in code yet but I think it could be built.
i've sorta made ad-hoc systems that do this myself, but didn't think to open source them
one thing that caught my eye was the part about incorporating feedback into the knowledge base - can you elaborate how you handle that?