Customers want smaller CDs, not iPods. So I think this statement is true if you are improving on an existing idea (such as castled). But if you’ve got a transformative idea, talking to customers probably doesn’t help as much
There is only one Steve Jobs. The rest of us are better off talking to customers first. That thinking (transformative idea... which generally amounts to nothing more than 'X of Y'), is why most startups fail.
True, from a numbers perspective most startups are a Y (in an X of/for Y) but every once in a while someone becomes an X (Uber, Airbnb, Dropbox) and that’s what’s exciting about reading a “Show HN” for me.
Hi HN! We are excited to share something we have been building for the last year.
Castled is a marketing automation platform built on data warehouses like Snowflake, BigQuery, Redshift, and Postgres. Castled allows you to directly use the user data in Snowflake and target them across channels like Email, Sms, WhatsApp, push, and In-app - without having to copy the data to another tool.
We would love you to try it out and give us your feedback. We have created a demo database and put some dummy data there so you can try us out without any hassles. If you have a google account, directly log in using https://app.castled.io/auth/login, or else use the signup link to log in.
Hi HN, We're Arun, Frank and Abhilash from Castled Data(https://castled.io). Castled is a marketing automation platform built directly on top of modern data warehouses like Snowflake, BigQuery, Redshift, and Postgres. Here is a quick demo:
I was just wondering if integrating with ClickHouse is on the roadmap? ClickHouse was built and literally designed for massive web event and click streams. We'd be happy to help, let us know and best wishes ahead
Hi HN, We're Frank, Abhilash, and Arun from Castled Data(https://castled.io). Castled is a marketing automation platform built directly on top of modern data warehouses like Snowflake, BigQuery, Redshift, and Postgres. Here is a quick demo:
We started our journey by building an open-source Reverse ETL solution to make the warehouse data actionable to marketers. However, after talking to 100s of marketers, we realised that modern B2C marketers needed to use billions of customer data points from the data warehouse to run personalised marketing campaigns. While Reverse ETL could sync all this data to marketing tools like Braze, Iterable, etc., the traditional marketing solutions were fundamentally not designed to store and process so much data.
The restrictions surfaced in the form of illogical data volume-based pricing and restrictive data retention - usually just a few months. Moreover, copying the data to multiple external tools created more complications w.r.t security audits(e.g., SOC2, GDPR, etc.) and other data privacy protocols. The data engineers also faced numerous issues maintaining the Reverse ETL pipelines, which often failed due to API timeouts, rate limits, etc.
That's when we changed course to build a warehouse-native marketing solution. Since we do not store any customer data with us, marketers could finally use all the data from the data warehouse to engage their customers without any tradeoffs. The headless approach also enabled us to offer marketers a more rational pricing model.
Marketing automation solutions have always been built on top of databases, which allowed transactional queries and point lookups. So solving the same use cases on top of a data warehouse that only supports analytical workloads would typically be a significant challenge. But luckily for us, 95% of the B2C marketing use cases required bulk access patterns, which data warehouses were uniquely designed to solve and even enabled us to offer capabilities that were not possible with traditional solutions.
One major challenge was supporting real-time campaigns, where we had to react to product events in sub-second latency using data from Snowflake. This required point lookups in Snowflake on each event, which was not scalable. We had to solve it using "contextual" data caches on our side. But we expect the advent of technologies like Unistore to make it easier to support use cases that require low-latency lookups on top of the data warehouse.
We have been building Castled for a year. Until now, we have enabled many businesses to solve their marketing data problems by replacing the combination of Reverse ETL + B2C marketing solutions with Castled. So, if you have a data problem in marketing, feel free to sign up and try our product for free - no credit card required. We have our solution hosted at https://castled.io.
Thats a great question! We dont use updated timestamps to compute deltas, as thats unreliable and can cause data loss depending on your transaction window.
We keep snapshot tables on your data warehouse(in our own custom schema, so that you dont have to provide Castled write access to any of your production schemas). The snapshot tables are then used as the baseline to compare the query results everytime the pipeline runs. Frankly, we have not really seen a use case of transferring 5 billion rows in a Reverse ETL pipeline. This is mostly because of the fact that our destination apps are mostly transactional systems and cannot really store so much of data. For example, salesforce destination can store max 10GB of data. Because of this, we are storing the actual tuple values in the snapshot. We have easily scaled our pipelines to compute deltas from queries which returns up to 100 million records. To optimize this further, we are also considering to keep the hashes of the tuple values instead of the actual values.
Yes, we need to know the primary key of your query results. This is required to handle failures and to remove the failed records from the snapshot table, so that those can be retried on the next pipeline run.
You mean the warehouse/app credentials, when you say sync configurations? If so, yes, that seems like a great idea. Infact I think your warehouse credentials are already there in your dbt repo in a specific format. Castled can directly read those credentials from there.
That is not what I meant, but also pretty interesting.
I actually mean the 'definition' of the syncs themselves.
I am picturing JSON or YAML that describes the source fields, their mapping to the destination fields, and any other meta about the sync: frequency, number of retries, whatever else that you could configure in the UI
So when I go and update my dbt model to modify one of the tables that I am syncing from, I can make the corresponding changes to my Castled settings file, and release it all as one atomic update to my data infrastructure.
It might be a small number of people who would want something like that, but it's definitely something I would have been excited about when I was running a data team.
We can definitely consider that. But I feel its a lot of config and can be error prone. For instance, source-destination field mapping configs might be complex and have various issues like data type mismatches, typos in field names etc and a user interface is better suited to guide you through the entire process.
But I see value in exporting the config to a github repo after the pipeline is created and thereafter future edits can be done via the github repo. Does that make sense?
Yes 100% -- you could also imagine just syncing from the UI to a repo, rather than trying to make the config human-editable. Toggle into a branch in the UI, make edits, and have those committed to the repo by the tool.
> It might be a small number of people who would want something like that, but it's definitely something I would have been excited about when I was running a data team.
Yeah this one certainly depends on the target customer. For me, any tool that didn't have source control integration for configuration would be a non-starter. But it's quite possible that the target audience for this tool doesn't even understand the term "source control".
At Grouparoo, this is a primary use case. We have a UI that engineers use locally. This helps gets things right. It outputs a JSON configuration that is checked in. When that is deployed, it does all the syncing.
A year back when we started to built Castled, this technology which syncs data from cloud warehouses to your operational tools did not have a name. The term "Reverse ETL" became popular somewhere in the beginning of 2021. We used this term, since we know that the data community knows this technology by this name now.
But my personal take is that "Reverse ETL" is still a new technology in the sense that it completes the modern data stack, which is built around cloud data warehouses.
