pytorch之加正则项

转：LinkL2 regularization：optimizer = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-5)lambd = torch.tensor(1.)l2_reg = torch.tensor(0.)for param in model.parameters():l2_reg += torch.nor

Adversity-sl

3127人浏览 · 2020-11-12 14:47:06

Adversity-sl · 2020-11-12 14:47:06 发布

转：Link
L2 regularization：

optimizer = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-5)

lambd = torch.tensor(1.)
l2_reg = torch.tensor(0.)
for param in model.parameters():
    l2_reg += torch.norm(param)
loss += lambd * l2_reg

L1 regularization：inclulde weight only

L1_reg = torch.tensor(0., requires_grad=True)
for name, param in model.named_parameters():
    if 'weight' in name:
        L1_reg = L1_reg + torch.norm(param, 1)

total_loss = total_loss + 10e-4 * L1_reg