SELU

When using kaiming_normal or kaiming_normal_ for initialisation, nonlinearity='linear' should be used instead of nonlinearity='selu' in order to get Self-Normalizing Neural Networks. See torch.nn.init.calculate_gain() for more information. SELU. Docs. Access comprehensive developer documentation for PyTorch.