Difference Between KL Divergence and Cross-Entropy

 

Difference Between KL Divergence and Cross-Entropy 

PointKL DivergenceCross-Entropy
1. MeaningMeasures the difference (information loss) between true distribution PP and model QQ.Measures the total coding cost of representing PP using QQ.
2. FormulaDKL(P∥Q)=∑P(x)log⁡P(x)Q(x)D_{KL}(P\|Q)=\sum P(x)\log\frac{P(x)}{Q(x)}H(P,Q)=−∑P(x)log⁡Q(x)H(P,Q)=-\sum P(x)\log Q(x)
3. RelationDKL(P∥Q)=H(P,Q)−H(P)D_{KL}(P\|Q)=H(P,Q)-H(P)H(P,Q)=H(P)+DKL(P∥Q)H(P,Q)=H(P)+D_{KL}(P\|Q)
4.InterpretationExtra error caused by using the wrong model.Total uncertainty/error when predicting with the model.
5. Use in MLUsed to compare distributions (e.g., variational inference, VAEs).Used directly as a loss function in classification problems.

Comments

Post a Comment

Popular posts from this blog

Advanced Mathematics for Computer Science HNCST409 KTU BTech Honors 2024 Scheme

About Me Syllabus Advanced Mathematics for Computer Science HNCST409 BTech Honors 2024 Model Question Paper and Answers University Question Paper HNCST409 BTech Honors-April 2026 and Answer Scheme Module- I Number Theory  Introduction Group , Ring and Fields Divisibility Modular Arithmetic and Congruences Equivalence Relation - Congruence Linear Congruences Solving simultaneous congruences- CRT Modular Inverses      Euclidean algorithm      Extended Euclidean algorithm  Euler’s and Fermat’s little theorem  Euler's Totient Function Prime Numbers and Prime-Power Factorization Fermat and Mersenne Prime Primality Testing and Factorization Wilson's Theorem Pseudo Primes and Carmichael Numbers Primality testing       Miller-Rabin      AKS Galois Field (introduction and basic operations) RSA cryptosystem: key generation - encryption/decryption  Hash functions (introduction) Module - II Optimiza...
Read more

Convex and Non Convex Sets

Image
Convex Sets Intuitive Idea A convex set is a collection of points such that the straight line segment connecting any two points in the set lies entirely within the set . Think of a solid ball or a solid square : no matter which two points you choose, the line between them never goes outside. A non-convex set has dents, holes, or gaps , so a line between two points may pass outside the set. Visual Intuition Convex → no dents or holes Non-convex → has indentations or holes Pic Curtesy: wikipedia Definition (Simple Terms) The Rule Pick any two points in the set. Draw the straight line segment between them. If the entire line stays inside the set , the set is convex. If this fails even once, the set is non-convex . Mathematical Definition A set C ⊆ R n C \subseteq \mathbb{R}^n   is convex if for any two points x 1 , x 2 ∈ C x_1, x_2 \in C  and for any number θ ∈ [ 0 , 1 ] \theta \in [0,1] , θ x 1 + ( 1 − θ ) x 2 ∈ C \theta x_1 + (1 -...
Read more

Equivalence Relation

  What Is an Equivalence Relation? Let A A  be a set. A relation \sim ∼ on A A  is called an equivalence relation if it satisfies three properties : 1. Reflexive For every a ∈ A a \in A , a ∼ a a \sim a 2. Symmetric For all a , b ∈ A a, b \in A , a ∼ b ⇒ b ∼ a a \sim b \Rightarrow b \sim a 3. Transitive For all a , b , c ∈ A a, b, c \in A , a ∼ b  and  b ∼ c ⇒ a ∼ c a \sim b \text{ and } b \sim c \Rightarrow a \sim c Why Equivalence Relations Are Important Equivalence relations allow us to: Group elements into equivalence classes Treat different objects as “essentially the same” Build structures like modular arithmetic , finite fields , and quotient sets Congruence Modulo n n n Let n ∈ Z , n > 0 n \in \mathbb{Z}, n > 0 . For integers a a  and b b , we say: a ≡ b ( m o d n ) a \equiv b \pmod{n} if and only if: n ∣ ( a − b ) n \mid (a - b) That is, a − b a - b  is divisible by n n . Claim Congruence modulo n n n is...
Read more