Difference Between KL Divergence and Cross-Entropy

 

Difference Between KL Divergence and Cross-Entropy 

PointKL DivergenceCross-Entropy
1. MeaningMeasures the difference (information loss) between true distribution PP and model QQ.Measures the total coding cost of representing PP using QQ.
2. FormulaDKL(P∥Q)=∑P(x)log⁡P(x)Q(x)D_{KL}(P\|Q)=\sum P(x)\log\frac{P(x)}{Q(x)}H(P,Q)=−∑P(x)log⁡Q(x)H(P,Q)=-\sum P(x)\log Q(x)
3. RelationDKL(P∥Q)=H(P,Q)−H(P)D_{KL}(P\|Q)=H(P,Q)-H(P)H(P,Q)=H(P)+DKL(P∥Q)H(P,Q)=H(P)+D_{KL}(P\|Q)
4.InterpretationExtra error caused by using the wrong model.Total uncertainty/error when predicting with the model.
5. Use in MLUsed to compare distributions (e.g., variational inference, VAEs).Used directly as a loss function in classification problems.

Comments

Post a Comment

Popular posts from this blog

Advanced Mathematics for Computer Science HNCST409 KTU BTech Honors 2024 Scheme

About Me Syllabus Advanced Mathematics for Computer Science HNCST409 BTech Honors 2024 Model Question Paper and Answers Module- I Number Theory  Introduction Group , Ring and Fields Divisibility Modular Arithmetic and Congruences Equivalence Relation - Congruence Linear Congruences Solving simultaneous congruences- CRT Modular Inverses      Euclidean algorithm      Extended Euclidean algorithm  Euler’s and Fermat’s little theorem  Euler's Totient Function Prime Numbers and Prime-Power Factorization Fermat and Mersenne Prime Primality Testing and Factorization Wilson's Theorem Pseudo Primes and Carmichael Numbers Primality testing       Miller-Rabin      AKS Galois Field (introduction and basic operations) RSA cryptosystem: key generation - encryption/decryption  Hash functions (introduction) Module - II Optimization Convex and Non Convex sets Convex Hull Identifying convex sets Applicatio...
Read more

Convex and Non Convex Sets

Image
Convex Sets Intuitive Idea A convex set is a collection of points such that the straight line segment connecting any two points in the set lies entirely within the set . Think of a solid ball or a solid square : no matter which two points you choose, the line between them never goes outside. A non-convex set has dents, holes, or gaps , so a line between two points may pass outside the set. Visual Intuition Convex → no dents or holes Non-convex → has indentations or holes Pic Curtesy: wikipedia Definition (Simple Terms) The Rule Pick any two points in the set. Draw the straight line segment between them. If the entire line stays inside the set , the set is convex. If this fails even once, the set is non-convex . Mathematical Definition A set C ⊆ R n C \subseteq \mathbb{R}^n   is convex if for any two points x 1 , x 2 ∈ C x_1, x_2 \in C  and for any number θ ∈ [ 0 , 1 ] \theta \in [0,1] , θ x 1 + ( 1 − θ ) x 2 ∈ C \theta x_1 + (1 -...
Read more

Gradient-Based Methods for Optimization-Gradient Descent Algorithm

  Gradient-Based Methods for Optimization (with Gradient Descent Algorithm) 1. Optimization Problem Setup In optimization, we aim to find min ⁡ x ∈ R n f ( x ) \min_{x \in \mathbb{R}^n} f(x) where: f ( x ) f(x)  is a real-valued objective function x = ( x 1 , x 2 , … , x n ) x = (x_1, x_2, \dots, x_n)  is a vector of variables In machine learning, f ( x ) f(x)  is typically a loss or cost function . 2. What is a Gradient-Based Method? A gradient-based method uses first-order derivative information (the gradient) to guide the search for an optimum. Key Idea The gradient ∇ f ( x ) \nabla f(x)  points in the direction of steepest increase The negative gradient points in the direction of steepest decrease 📌 Hence, to minimize a function, we move in the negative gradient direction . 3. Geometric Interpretation of the Gradient The gradient is perpendicular to level curves Moving along − ∇ f ( x ) -\nabla f(x)  decreases th...
Read more