pytorch自己实现一个CrossEntropy函数

This article was original written by Jin Tian, welcome re-post, first come with https://jinfagang.github.io . but please keep this copyright info, thanks, any question could be asked via wechat: jintianiloveu

理论(theory)

$$CrossEntropy = \sum{(ylog(y’) + (1-y)log(1-y’))}$$

[[1, 0, 0],
[0, 1, 0]]

[[545, 54, 2],
[232, 54, 546]]


1. 先计算softmax, 然后计算log;
2. 计算 $yi*log(yi)$

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50  import torch import numpy as np # label = torch.Tensor([ # [1, 2, 3], # [0, 2, 1], # ]) label = torch.Tensor([0, 2, 1]).long() fc_out = torch.Tensor([ [245, 13., 3.34], [45., 43., 37.], [1.22, 35.05, 1.23] ]) def one_hot(a, n): b = a.shape[0] c = np.zeros([b, n]) for i in range(b): c[i][a[i]] = 1 return np.array(c) def softmax(a): return [np.exp(i)/np.sum(np.exp(i)) for i in a] def cross_entropy_loss(out, label): # convert out to softmax probability out_list = out.numpy().tolist() out1 = softmax(out_list) print(out1) out2 = torch.softmax(out, 0) print(out2) # [0, 2, 1] -> [[1, 0, 0], [0, 0, 1], [0, 1, 0]] # onehot label and rotate label_onehot = one_hot(label, 3) loss = np.sum(out1 * label_onehot.T) print(loss) loss = torch.nn.CrossEntropyLoss() lv = loss(fc_out, label) print(lv) lv = cross_entropy_loss(fc_out, label) print(lv)