https://twitter.com/irinarish/status/1488767050126397444



역전파 알고리즘


결과물을 보고 역으로 신경망을 거꾸로 거슬러올라가며 수정하는 알고리즘인데

뇌에서 있다 없다 말이 많고 대체로 없다는 쪽이 많아보임


그런데 이번에 역전파 알고리즘이 아닌

Difference-Target-Propagation (DTP) 차이-표적-전파 알고리즘이라는

새로운 알고리즘이 나왔는데 생물학적으로 보다 인간 뇌에 가까우면서도

역전파 알고리즘보다 성능이 떨어지지 않는 동급의 성능을 보였다고 함


DTP는 피드백이 아니라 피드포워드로 에러 그래디언트가 아니라 표적 값을 전파하는데

이게 보다 인간 뇌에 가깝다고 함


In this paper, we propose to revisit the GN interpretation of

DTP by having the feedback pathway synaptic weights 

compute layer-wise BP targets rather than GN targets. To this

end, we propose a novel feedback weights training scheme

which, by construction, pushes the Jacobian of the feedback

operator towards the transpose of its feedforward counterpart, 

in a layer-wise fashion and without having to use direct

feedback connections in the feedback pathway. Therefore,

assuming this condition holds for all the layers and keeping 

everything else unchanged in the DTP algorithm, the

DTP feedforward weight updates closely approach those

of BP. This leads us to a scalable biologically plausible

approximation of BP


• We propose a novel Local Difference Reconstruction

Loss (L-DRL) along with an algorithm to train the

feedback weights which ensures that the Jacobian of

the feedback pathway matches the transpose of the

Jacobian of the associated feedforward pathway (Section 4.1, Theorem 4.2, Alg. 3). 

We call this condition the Jacobian Matching Condition (JMC, Definition 4.1).


• Assuming the JMC holds for a given architecture and

using the standard DTP equations to propagate targets,

we demonstrate that DTP feedforward weight updates

approximate BP gradients (Section 4.2, Theorem 4.3).

We say that such an architecture satisfies the Gradient

Matching Property (GMP).


• We numerically demonstrate the GMP and JMC, 

showing that L-DRL is more efficient than DRL to align

feedforward and feedback weights and that the GMP is

subsequently significantly better satisfied (Section 5.1-

5.2).


• Finally, we validate our novel implementation of DTP

on training experiments on MNIST, Fashion MNIST,

CIFAR-10 and ImageNet 32×32 (van den Oord et al.,

2016) (Section 5.3). In particular, we achieve a 89.38

% accuracy on CIFAR-10 and 60.6 % top-5 accuracy

on ImageNet 32×32, which are the best performances

ever reported in the DTP literature on these datasets

and nearly match the performance of BP on the same

architectures.