From 399bb9515a1222eb82e7329a170c6efa174f9fd8 Mon Sep 17 00:00:00 2001
From: Rain&Bus <rainandbus@gmail.com>
Date: Mon, 29 Apr 2024 18:59:38 +0800
Subject: [PATCH] vault backup: 2024-04-29 18:59:38

---
 Record/DL/Loss.md | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/Record/DL/Loss.md b/Record/DL/Loss.md
index 1276eee..8e7ff68 100644
--- a/Record/DL/Loss.md
+++ b/Record/DL/Loss.md
@@ -1 +1,13 @@
+# 熵
+随机变量$X = \{x_1,x_2,...,x_i\}$，对应的概率为$p_i = p(X = x_i)$，则熵为
+$$
+H(X) = - \sum_{i=1}^{n}p(x_i) \log p(x_i)
+$$
+> $p(x_i)=0$时，$p(x_i)logp(x_i)=0$
+
+# 交叉熵
+$$
+H(p,q) = \sum_x p(x) \frac{1}{q(x)}=-\sum_x p(x) \log q(x) 
+$$
+> 表示对预测分布$q(x)$使用真实分布$p(x)$来进行编码时所需要的信息量大小。
 # KL散度