I am a fan of the thread-per-core model, but I do not share their views on the data sharding per core
While it increases the data locality, I have seen a few software following this sharding model (notably scylla) that work really bad once the load is not evenly distributed across all shards
When that happens it can be a huge waste of resources and can give lower performance (depending on the type of load)
Imho unless you are absolutely sure about the type of load, leave the sharding to dividing data between servers, or have some mechanism that can shift to sharing the load between threads if the system imbalance is too great
> While it increases the data locality, I have seen a few software following this sharding model (notably scylla) that work really bad once the load is not evenly distributed across all shards
Serial access to hot keys is a hard thing to design around, and you're right that sharding doesn't solve that problem. Worse, it exposes other keys that just happen to share the shard (or the core) to poor performance.
There are a couple of well-understood solutions to this problem. The obvious one is to dynamically re-balance shards based on heat, either moving some keys or split/merge. This is the same tactic that many distributed databases use, and while it's complex to do, it's easier to do locally than distributed. Another option is stochastic re-balancing, like the Stochastic Fairness Queuing (https://ieeexplore.ieee.org/document/91316) model used in networking. Here, shards are randomly re-shuffled occasionally. Doesn't fix the noisy neighbor problem, but does mean that the noisyness moves around. That might seem silly, but it's pretty much what's going to happen under the covers of the non-sharded version of the code when the scheduler gets involved, only easier to reason about.
> When that happens it can be a huge waste of resources and can give lower performance (depending on the type of load)
Lower apparent performance for neighbors of the heavy hitter, sure. Under which other circumstances does it reduce performance?
> Imho unless you are absolutely sure about the type of load, leave the sharding to dividing data between servers, or have some mechanism that can shift to sharing the load between threads if the system imbalance is too great
I'm a bit puzzled by this. Distributed systems have exactly the same problem, and solving that problem is much harder there because the cost of contention is higher, data movement is more expensive, and you have to deal with a lot more failure cases.
The statistics may be better for distributed systems because a hot tenant has to be a lot hotter to make hot box than a hot core. But that's a very specific kind of bet, and if you end up with a tenant that does cause a hot box you have an even harder problem to solve.
Dynamo (https://www.allthingsdistributed.com/files/amazon-dynamo-sos...) solves this by moving the key space around, as do many similar kinds of systems. It's not easy, though, and filled with caveats. If you're scared of sharding, distributed sharding should be scarier than on-box sharding.
There's a few mitigations that can be done for poorly-sharded data. Observability being the first step in the process. We've added a toppartitions method to nodetool to spot such monsters in the wild. The idea that you can just autobalance out of poor sharding methods is difficult in practice. Your "hot partition" could be because of a stochastic event ("the dress that broke the Internet") vs. a perennially "hot key" that, no matter where you place it, it's going to be hot.
I've been building data sharding architectures for at least a decade, so the problems you raise are ones I am intimately familiar with. Many of the data models I've worked with have unavoidably unpredictable dynamic distributions of both data and load, so it is a problem that can't be ignored. You are correct that this will essentially break naive data sharding architectures.
It is possible to design data sharding architectures where balancing of both data and load across cores is continuous and smooth, with surprisingly minimal overhead and coordination cost. In fact, there are multiple ways of doing it, depending on your workload characteristics. At this point, the design idioms for this style of architecture are refined and robust, so there is no computer science reason the problems you raise need to exist in a real implementation. The reason it seems "difficult" in practice is because so many designs insist on loosely coupling and weakly scheduling storage, execution, network, etc. If your architecture concept is slapping a thin layer on top of RocksDB, it won't be feasible. Every part of the stack needs to understand and be designed to the model. The end result is actually quite elegant in my view, and with unmatched throughput.
People design distributed systems with simple static sharding schemes because they are obvious, easy, and it lets you cut a lot of corners on the rest of your system design. It is not the only way to design a distributed system, continuous adaptive resharding and load shedding is a demonstrably viable option, and it is much easier to implement within a single server than on an actual cluster of networked computers.
Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases
"Since our system has a high tolerance to failures, we can leverage this for maintenance operations that cause segment unavailability. For example, heat management is straightforward. We can mark one of the segments on a hot disk or node as bad, and the quorum will be quickly repaired by migration to some other colder node in the fleet."
Shard per core has challenges. Scylla solves it partially by making the client shard aware (topology aware), so the client submits requests directly to the right cpu core using a different port. This way no data gets blocked due to hot keys or imbalance in other shards. You do need to have a good key distribution. We're implementing a new mechanism to split key ranges dynamically. It will be a fast an parallel mechanism.
If the number of connections is not small, Scylla will crank up any other implementation with traditional locking.
While it increases the data locality, I have seen a few software following this sharding model (notably scylla) that work really bad once the load is not evenly distributed across all shards
When that happens it can be a huge waste of resources and can give lower performance (depending on the type of load)
Imho unless you are absolutely sure about the type of load, leave the sharding to dividing data between servers, or have some mechanism that can shift to sharing the load between threads if the system imbalance is too great