bitcoin
bitcoin

$64702.434704 USD

0.16%

ethereum
ethereum

$1814.872353 USD

-0.75%

tether
tether

$0.998909 USD

-0.02%

bnb
bnb

$592.426041 USD

-0.03%

usd-coin
usd-coin

$0.999890 USD

0.01%

xrp
xrp

$0.991637 USD

3.32%

solana
solana

$75.072867 USD

3.15%

tron
tron

$0.309677 USD

1.12%

hyperliquid
hyperliquid

$68.997207 USD

-0.94%

zcash
zcash

$954.829619 USD

3.54%

monero
monero

$505.913251 USD

1.66%

dogecoin
dogecoin

$0.059515 USD

1.02%

unus-sed-leo
unus-sed-leo

$9.291045 USD

-0.21%

chainlink
chainlink

$9.270670 USD

1.14%

cardano
cardano

$0.154129 USD

0.59%

Cryptocurrency News Video

Learn Deep Learning #25: Positional Encoding: Why Attention Needs to Be Told Where It Is

Nov 19, 2026 at 01:00 am QED Realized

Episode 23 derived attention and proved it is permutation-equivariant: shuffle the rows of the input and the rows of the output shuffle the same way, nothing else changes. That is a feature for a set and a disaster for a sentence, because "the cat sat" and "sat the cat" become the same bag of vectors. This episode starts by showing that failure with numbers, on a four-token, four-dimensional toy: the attention output on the permuted input is exactly the permuted output, to the last decimal. Then it states what a position code has to do, and works through the three ways the field has answered: a learned table added to the input (2018), the sinusoidal code of 2017, and rotary embeddings (2021), with ALiBi and learned relative biases alongside. The sinusoidal code gets the full treatment. The d equals 8 table for positions 0 to 7 is computed and shown, the geometric spread of wavelengths from 2 pi to about 10,000 times 2 pi is explained as a binary counter made continuous, and the theorem that justifies the sines and cosines is derived: for each frequency pair, moving forward k positions is a fixed 2 by 2 rotation that depends on k alone, so relative offsets are linearly recoverable, and the dot product of two codes depends only on their separation. Rotary embeddings are derived in the complex plane, where the same idea lands directly on the attention score. The toy is rerun with a position code, and the permuted output is no longer a permuted copy. The last slide separates what is a theorem from what is an empirical comparison, including the honest state of length generalisation. 📚 Continues in: → 'Autoencoders and the Bottleneck as Compression' — the next architecture is not about sequences at all: a network forced through a narrow middle, and what the narrowness makes it learn. 🎓 Best for: Programmers, scientists and mathematicians who have used a deep learning framework and want to know what it is actually computing — and students who want the theory before the tooling. Comfort with vectors, matrices, and single-variable calculus is assumed; probability, matrix calculus and optimisation are built here from scratch. — QED Realized — Clean explanations of the ideas that take a real lecture to land. 🔔 Subscribe so you don't miss the next one: https://www.youtube.com/@QEDRealized ❤️ Support me on Patreon: https://www.patreon.com/QEDRealized These courses are free and they stay free. Patreon is how you can support me directly and help fund more of them — every pledge moves this closer to being full-time work. #DeepLearning #NeuralNetworks #MachineLearning #Transformers #MachineLearningMath — Tags (for search) — positional encoding explained, why transformers need positional encoding, attention permutation equivariance proof, sinusoidal positional encoding derivation, sin cos positional encoding rotation matrix property, positional encoding dot product depends on offset, geometric wavelengths 2 pi to 10000 2 pi, binary counter analogy positional encoding, learned absolute position embeddings cannot extrapolate, rotary position embedding rope derivation complex numbers, rope relative position property, alibi linear bias attention, relative position bias per offset, length generalisation transformers empirical, four token attention toy with and without positional encoding, transformer positional encoding history 2017 2018 2021, deep learning course episode 25, positional encoding math derived ⏱ Chapters 0:00 What today builds 1:08 The theorem from episode 23, restated 2:34 Four tokens, every number 3:57 Four requirements, and three candidates that fail them 5:11 The simplest injection, and its limit 6:24 The definition and every number 7:48 Why the frequencies are a geometric ladder 8:59 Why sines and cosines, derived 10:39 The offset-only dot product, derived and plotted 11:58 The same four tokens, now with position 13:26 Put the rotation on the score itself 15:08 Bias the score by the offset 16:21 Epistemic status, and five dates 18:00 Recap 19:18 Thanks for watching
Video source:Youtube

Disclaimer:info@kdj.com

The information provided is not trading advice. kdj.com does not assume any responsibility for any investments made based on the information provided in this article. Cryptocurrencies are highly volatile and it is highly recommended that you invest with caution after thorough research!

If you believe that the content used on this website infringes your copyright, please contact us immediately (info@kdj.com) and we will delete it promptly.

Other videos published on Nov 19, 2026