add Chinese annotations to all source files for learning purposes
Annotated 16 source files covering the full architecture: engine (scheduler, block manager, model runner), layers (attention, linear, sampler, etc.), model (qwen3), and utils. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -4,6 +4,12 @@ import torch.nn.functional as F
|
||||
|
||||
|
||||
class SiluAndMul(nn.Module):
|
||||
"""SwiGLU 激活函数:SiLU(gate) * up。
|
||||
|
||||
输入是 gate 和 up 拼接的张量,沿最后一维一分为二,
|
||||
对前半部分应用 SiLU 激活后与后半部分逐元素相乘。
|
||||
这是 LLaMA/Qwen 系列模型中 MLP 层的标准激活函数。
|
||||
"""
|
||||
|
||||
@torch.compile
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
|
||||
Reference in New Issue
Block a user