You don't have to do it manually.
There might be some numbers you're missing when summing it up.
Pure PyTorch implementation of Fast Differentiable Sorting and Ranking (Blondel et al.). Much of the code is copied from the original Numpy implementation at google ...