Madoff's funds weren't exactly exchange-traded, generic investment vehicles. As mentioned in the article, part of the money is clawed back from "feeder funds". Those were sales or regulatory vehicles that passed 100% of their capital on to Madoff.
As to your examples:
Person X is irrelevant. You claw back the money from fund Y. Fund Y may, in turn, have arrangements with person X requiring them to return any money, but that's actually tangential. (this is like 1st-semester law again, yeay!)
Person A owes <total sum taken out> - <total sum put in> (this is like 10th grade math again. yeay!)
So Person X makes off as a pure winner in the clawback (doesn't lose Fund Y profits, gets "made whole" of direct losses in clawback) Not "fair" but "legal". Yay!
In real-world experience, rather than 10th-grade story problems, the devil is in the details. Dates are important (you're eligible for recovery of losses from date range, you're liable for recovery from other date range), and Person A is likely to be vulnerable to clawback of earlier profits, even if reinvested and later lost. The profits from the scheme are separate transactions from the later losses in the same scheme. In this case, your simple math would be more fair, but probably not how the legal system would work. Yay!
You can see examples of this in capital gains taxes on people who made and lost a lot in cryptocurrency boom/bust cycles. Arbiters may have a lot of discretion in evaluating claims in these big cases, but I'd rather have the law on my side than relying on the discretion of an arbiter.
Sure, that simplifies things and is how the law treats it. But there is still complexity. You have some cases where a single true owner invested via a family office, via different trusts, via feeder funds all at different times with different amounts taken out and different profit and losses. Sometime the profits from one venue are compensation to another. In these scenarios, you can treat them all as separate cases, or you can pull them together and negotiate an umbrella settlement. My understanding is that this has been done when possible, including with some hedge-funds who have purchased a variety of positions and even feeder funds themselves.
As to your examples:
Person X is irrelevant. You claw back the money from fund Y. Fund Y may, in turn, have arrangements with person X requiring them to return any money, but that's actually tangential. (this is like 1st-semester law again, yeay!)
Person A owes <total sum taken out> - <total sum put in> (this is like 10th grade math again. yeay!)