There are several useful curves that keep coming up in machine learning so here we will plot them and consider their properties.
Constrained on one side:
- ReLU
- Softplus
- Exponential
- Leaky ReLU
- Swish & GELU
- Log
Bounded on both sides:
- Logistic
- Gaussian CDF
- Tanh
- Cosine
- Generalized logistic