I published a video that explains Self-Attention and Multi-head attention in a different way -- going from intuition, to math, to code starting from the end-result and walking backward to the actual method.
Hopefully this sheds light on this important topic in a way that is different than other approaches and provides the clarity needed to understand Transformer architecture. It starts at 41:22 in the below video.
This video tutorial provides an intuitive, in-depth breakdown of how an LLM learns language and uses that learning to generate text. Key concepts shown below are covered in a way that is both broad and deep, while still keeping the material accessible without losing technical rigor:
* Historical context for LLMs and GenAI
* Training an LLM -- 100K overview
* What does an LLM learn during training?
* Inferencing an LLM -- 100K overview
* 3 steps in the LLM journey from pre-training to serving
* Word Embeddings -- representing text in numeric format
* RMS Normalization -- the sound engineer of the Transformer
* Benefits of RMS Normalization over Layer Normalization
* Rotary Position Encoding (RoPE) -- making the Transformer aware of token position
* Masked Self-Attention -- making the Transformer understand context
* How RoPE generalizes well making long-context LLMs possible
* Understanding what Causal Masking is (intuition and benefit)
* Multi-Head Attention -- improving stability of Self Attention
* Residual Connections -- improving stability of learning
* Feed Forward Network
* SwiGLU Activation Function
* Stacking
* Projection Layer -- Next Token Prediction
* Inferencing a Large Language Model
* Step by Step next token generation to form sentences
* Perplexity Score -- how well did the model does
* Next Token Selector -- Greedy Sampling
* Next Token Selector -- Top-k Sampling
* Next Token Selector -- Top-p/Nucleus Sampling
* Temperature -- making an LLM's generation more creative
* Instruction finetuning -- aligning an LLM's response
It is a shame that most people are unaware of the efficacy of Homeopathy to treat ADD, ADHD and autism-spectrum disorders. Do some research and you'll see how hundreds of thousands of people have been cured of these ailments using Homeopathy alone.
Do some research and you'll find homeopathy is bunk. There's no science behind it, studies have shown it completely ineffective, and besides, it's just clean water.
You are saying this out of the research you have done yourself, right? And yes, I have done the research and also have experienced it first hand -- something that Stanford doctors could not do (and an army of 3 different specializations nonetheless) was cured using Homeopathy. Is this placebo effect?
With the rampant abuse of market power by the drug companies with Ritalin and anti depressants, I wonder if homeopathy could be a helpful competitor. Magic water can't cause harm and takes some money away from you so you can't use it for something harmful.
I don't remember the names, but a whole generation of anti depressants was shown to be completely useless and harmful. The initial studies were fabricated and/or got into journals unethically. The various decade long studies on Ritalin also show it to be largely useless and harmful.
Homeopathy is offensive to people who respect science, but it's a noble lie in the current market conditions.
If that's the case then I disagree with your characterization; I don't see any passage that refers to decade long studies that show that Ritalin is "harmful" in the long-term.
What would be the licensing model if the product is open source, but is NOT free (i.e., customers are charged for an open-source product in which BDB is used)?
Hopefully this sheds light on this important topic in a way that is different than other approaches and provides the clarity needed to understand Transformer architecture. It starts at 41:22 in the below video.
https://youtu.be/6jyL6NB3_LI?t=2482