There are have been attempts at "phase tree" filesystems with essentially transactional write semantics for data and metadata simultaneously. Tux2 was one such nascent attempt that ended (rightly or wrongly) due to patent conflict with Netapp's WAFL [1].
The interesting thing is, I just took a look at what I believe were the patents in conflict, and they appear to have expired in 2015 [2,3]. Combining the phase tree concept with, say, Merkle trees might open up very interesting possibilities for large-scale reliable file storage.
Windows has transactional NTFS. It was never widely adopted, in part because it requires use of a whole different filesystem API (CreateFileTransacted instead of CreateFile, etc). I think, if it had been more seamless, it may have been more successful. Given it is rarely used, Microsoft has marked it as deprecated for potential removal in future Windows versions.
Iirc it was implemented with a pretty seamless API at the undocumented ntdll layer, which let you set a current transaction on a handle then do API calls as usual.
The issue is that while NT mostly reasons about files as handles, Win32 deals primarily with the name for many operations. So they duplicated every api that took a filename.
Doesn't surprise me to learn that the NT-level API is nicer.
The NT API is in general much better thought out than the Win32 API. It is a real pity that Microsoft chose to leave the former largely undocumented and tell people to use the later instead.
