It's ever discussed in https://github.com/apache/arrow-rs/issues/3888 for compar... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		tison on March 10, 2024 \| parent \| context \| favorite \| on: S3 is files, but not a filesystem It's ever discussed in https://github.com/apache/arrow-rs/issues/3888 for comparing object_store in Apache Arrow to the APIs provided by Apache OpenDAL. Briefly, Apache OpenDAL is a library providing FS-like APIs over multiple storage backends, including S3 and many other cloud storage. A few database systems, such as GreptimeDB and Databend, use OpenDAL as a better S3 SDK to access data on cloud storage. Other solutions exist to manage filesystem-like interfaces over S3, including Alluxio and JuiceFS. Unlike Apache OpenDAL, Alluxio and JuiceFS need to be deployed standalone and have a dedicated internal metadata service.

Lucasoato on March 10, 2024 [–]

I'm not sure if Alluxio could be substituted by OpenDAL as a local cache layer for TrinoDB.

tison on March 11, 2024 | [–]

If I get "local cache layer" correctly, it's possible. And it's even desired if you want to reduce the deployment burden.

Here are some related codes on how we implement such a layer in GreptimeDB:

* https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/o... * https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/m...

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact