Either that or an unfortunate accident

Interesting to see how the data transformation space evolves.

This week dbt acquired SDF (conversation here: https://news.ycombinator.com/item?id=42697764), and now Tobiko acquires Quary, another data tool built in Rust.

Excited to see how this will impact the compile times (and developer experience in general) of sqlmesh

Checking their own post, https://www.getdbt.com/product/dbt-core-vs-dbt-cloud, stuff such as the semantic layer and column-level lineage are Cloud exclusive

those were never in the core though.

Spanish I'd say

Really like the concept here!

Can't help to think that that if a Fivetran sync fails, the problem would come back.

Maybe a more sensitive solution is to take the last modified date of all of the sources and use the oldest one as the cutoff, that way you wouldn't be afraid of your relationship tests failing if the sync failed too, although I reckon that this solution assumes that your sync frequency is equal for all of your sources

Author here- thanks!

You are totally correct, if Fivetran fails on any of the tables then we'd likely see issues in the tests.

My thinking was this was ok, as I'd prefer my tests to fail than me blindly continue with missing data. Obviously a more sophisticated solution would be to page directly on this, but in the name of minimalism, it seemed like an ok solution.

There's also a Fivetran audit table that I considered using, as it contains data like this:

``` { "id": "9b21ce1e-6889-4e3f-ab8f-889425d1a9dd", "_fivetran_synced": "2022-02-04 00:11:27.413 UTC", "done": "2022-02-04 00:10:57.690 UTC", "message": null, "progress": "2022-02-04 00:10:55.757 UTC", "rows_updated_or_inserted": "3", "schema": "core_production_public", "start": "2022-02-04 00:10:57.576 UTC", "status": "Ok", "table": "incidents", "update_id": "013c201d-7fca-44cb-a5ee-93ba3e74ae5e", "update_started": "2022-02-04 00:10:55.756 UTC" }, ```

But figured that would have the same issue. That said, I think this is a valid approach for anyone with the more sophisticated alerting!

Imba looks great! Now that v2 is out, what would you say are your priorities regarding Imba? Is there a roadmap somewhere?

What are your thoughts about Gleam lang? Don't you think it covers those conditions?

It runs on top of Erlang if I see this right which isn't quite what I personally would be looking for (rust or MLTon, OCAML compile to native code).