Agreed. I was just reacting to the parent's comment that
> Non-differentiable here doesn't mean actually non-differentiable in the mathematical sense, it just means that the function does not expose a derivative that is accessible to you.
I read that as meaning that the loss functions being considered were differentiable in the mathematical sense, it was just hard to calculate the derivative.
My point is that the parent's comment is mostly right: a frequent challenge in ML is black box computational units which don't expose the necessary information to run autograd. Even if the underlying mathematical function is not differentiable everywhere, but only on most points, having autograd available is valuable for use in training.
Hence you get works like this [1] which reimplement existing systems in a way that is amenable to autograd.
[1] https://arxiv.org/abs/1910.00935
> Non-differentiable here doesn't mean actually non-differentiable in the mathematical sense, it just means that the function does not expose a derivative that is accessible to you.
I read that as meaning that the loss functions being considered were differentiable in the mathematical sense, it was just hard to calculate the derivative.