Unfortunately its not true in many cases I have seen, essentially because the tail of the error does not fall fast enough. Technically, for all bounded RVs all moments are bounded, but in some of these situations the variance is so high that its infinite for practical purposes.
All you need is the tail to fall slower than a quadratic.
In practice, essentially always, for real random variables X, Y, all of the following exist and are finite:
E[X], E[|X|], E[X^2], E[XY]
Var(X) = E[(X - E[X])^2]
Std(X) = Var(X)^(1/2)
Cov(X,Y) = E[(X - E[X]) (Y - E[Y])]
Cor(X,Y) = Cov(X,Y)/(Std(X) Std(Y))
E[Y|X]
E[X] and Std(X)are useful quite broadly and that we find/estimate them does not mean that we are working with a Gaussian distribution.
If the above is not true, then there is something bizarre and/or pathological, the ultimate edge case, and we need to review what we are doing.