Welcome to Rational Activations’s documentation!

Rational Activations provided in this package are learnable rational activation function to create rational neural networks. So far, only the pytorch and MXNET implementation are available.

Requirements:

This project depends on:

• pytorch

• numpy, scipy (if you want to add different initially approximated functions)

• matplotlib (if you want to use the plotting properties)

• torchvision (if you want to convert a torchvision model architecture to Rational)

Download and install: You can download from the Github repository or:

pip3 install rational-activations

To use it:

from rational.torch import Rational

rational_function = Rational() # Initialized closed to Leaky ReLU
print(rational_function)
#    Pade Activation Unit (version A) of degrees (5, 4) running on cuda:0
# or Pade Activation Unit (version A) of degrees (5, 4) running on cpu

Tutorials:

• Make a rational Network in Pytorch
• Find weights for Rational approximating new functions
• Visualise and debug the functions
  • Visualise the functions’ evolutions through learning
  • Use a tensorboardX SummaryWriter to plot the evolutions
• Apply the functions to torchvision model

Indices and tables

• Index

• Module Index

• Search Page

Cite us in your paper

Rational activations were first introduced as Padé Activation Units in [MSK19], and later developed as Rational Activation in [DSMK21].

To cite the package only: [DSM+20].

DSM+20

Quentin Delfosse, Patrick Schramowski, Alejandro Molina, Nils Beck, Ting-Yu Hsu, Yasien Kashef, Salva Rüling-Cachay, and Julius Zimmermann. Rational activation functions. https://github.com/ml-research/rational_activations, 2020.

DSMK21

Quentin Delfosse, Patrick Schramowski, Alejandro Molina, and Kristian Kersting. Recurrent rational networks. arXiv preprint arXiv:2102.09407, 2021.

MSK19

Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Padé activation units: end-to-end learning of flexible activation functions in deep networks. In International Conference on Learning Representations. 2019.