Do you have any insight into comparing
SurrealDB (https://surrealdb.com/docs/introduction/start) to DuckDB (https://shell.duckdb.org/)? I haven't done that comparison yet, I might now based on your recommendation. I saw DuckDB at Data Day Texas 2023 and their demo was intriguing. That's even for our use case which was not very SQL-y.
This is a fantastic take (seriously, no sarcasm here). I'll add that serverless platforms have not actually shown the ability to scale _efficiently_. It feels like every software architect has been talking about the 2023 March post^ from Amazon Prime Video on migrating from serverless to serverful. That is a direct response to the inability of serverless to scale because under the hood it is serverful but limited. Besides AWS Lambda and Google Cloud Run others have tried OpenFaaS and Cloudflare Workers. I think only Cloudflare Workers have "scaled" but that's because it's limited scale for each service at the edge with lots of edge devices.
I do not know if this is solvable: a permanent service with light autoscaling handles even inconsistent load so well and reduces operational complexity so much that I don't know if serverless will be anything more than toys. It is no coincidence that AWS Lambda's first language was a frontend language* (Node.js). Those use cases have low to moderate scale (because after that you split frontend and backend).
I agree with some of the points you've mentioned, but I also think that there definitely are great use-cases for AWS Lambda, and that they are (especially when using a good deployment tool) a great option for many different types of applications.
The biggest problem is that the computing model, or the architecture that is commonly required for this sort of apps (event-driven and asynchronous) is hard to develop, manage and reason about (especially with the async-lambda integrations that by default retry their work if they fail). This means that you need to think about them and architect them in a way that makes them idempotent.
If you configure everything properly, event-driven, AWS Lambda-based apps can be extremely useful. I just don't believe that the complexity required to add them to your application is worth the complexity that needs to be added to your app to make it all work.
What I wanted to say is that Lambda functions have their use-cases, and that if used reasonably, they can heavily simplify the application that they are used within.
Our product (https://stacktape.com, disclosure: Im a founder), makes it easy to deploy both container-based (ECS Fargate and ECS EC2) apps, and lambda functions, with almost no changes in the configuration). This allows our users to always choose the right technology/computing model for the use-case they are working on.
- Apache Spark starts its relatively slow decline in favor of vendor and in-house solutions.
- Cost drives cloud systems away from managed solutions (think AWS Batch) towards semi-managed solutions (think self-run k8s deployments but on AWS EKS).
- There are no improvements in autoscaling k8s clusters.
- Integration testing is replaced by testing in the development environment.
Kind of funny about that last bit of advice, as that is the opposite of the advice for other creative endeavours. In https://savethecat.com/ the book talks about telling your story ideas to everyone so that your creative juices get excited.
I do think you're right about tech. Try it yourself, build it out, have some fun, be a dork. Then tell people.
I think the programming working model will change as it did years ago when we moved from punch cards to files and then from assembly to higher-level languages. Programmers will continue to program solutions with assistance from code automation for less business-focused parts of the problem at hand. Consider the more analog version of this: nowadays we spend more and more time copying answers from StackOverflow than we do debugging at a lower level.
