As someone running a federated server (mastodon); you get more problems with local spam than with foreign spam.
Most instances will quickly shut down any spam, so it doesn't really fill the federated timeline that much. Local accounts are something you as an administrator need to act on, so they are the annoying ones.
In my experience, instances that allow spam are rare and the get quickly blacklisted by everyone from showing up in the timelines (there is luckily a neat middleground between blocking the instance entirely and not doing that; allow local users to subscribe to that instance, which is my default action for all instances that aren't hosting illegal material for me).
It doesn't look like that might scale at all. In other words, it works because Mastodon is a very small, niche community.
Good thing is that the Mastodon instance owners use shared block lists for servers that are currently used to ban free speech instances; that infrastructure could also be used to create a collectively managed list of instances that are full of spam.
Generally not, proactive blocking or silencing is reserved for instances that host illicit content like child porn or hate speech. Inter-moderator communication happens between your instance admin/democracy (or whatever structure your instance uses) and other instances.
Traditional tools like muting a user, silencing an instance (if they're spamming up the Federated Timeline on other instances, like Humblr.social does) and blocking (if the mods are total shit) can still be used. Small instances are the bulk of the content, so you aren't about to block them. Large instances like Mastodon.social are generally blocked tho (if your admin/democracy cares about a vibrant Fediverse).
Honestly, I think this is the wrong way of looking at things. The Fediverse isn't really like a single social network that's shattered amongst many instances ... it's more like a bunch of individual, separate hangouts, in the vein of classic forums and IRC chatrooms and fandom websites, which have the ability to talk to each other and share stuff if they want.
It's not so much about "blocking" mastodon.social as not choosing to link up with it - because your instance is a specific community, and actual human social relations thrive when they can choose who they're hanging out with. So you choose to talk to other smaller, more diverse communities that you can actually get to know a good chunk of the people in them, instead of just drinking straight from the firehose that is a big instance.
Federating with another instance is basically two groups of people deciding to hang out together, effectively merging. Federating with mastodon.social means, effectively, merging your community with mastodon.social ... which means submerging it beneath the mass of mastodon.social's much greater activity.
A vibrant Mastodon isn't a more-obtuse recreation of the Twitter experience. Twitter is already Twitter. It's about fostering all the things that Twitter isn't - personal and Dunbar-scaled and locally moderated and diverse.
I think this is inherently a sociological problem than a technological one. When there are enough people in an organically grown group, the community will become unsustainable and then 1) it disintegrates into multiple groups until it becomes stable again or 2) it transforms into a hierarchial, authoritative society where a small group of people makes most of the decisions.
Also, I think Mastodon’s design deliberately steers instances away from becoming large ones; it rather tries to be a nurturer for various small communities and subcultures. It doesn’t view size as a good thing, and that’s totally okay. Nowadays, I think we need less monoculture and more diversity/creativity in our society.
It'll probably end up "solved" in a similar way to email span being "solved" - if global spam starts to become a problem, you'll have big blocklists of spammy instances and general suspicion towards federating with any new instance that doesn't look sufficiently trustworthy.
Malicious actors definitely try to spam and data mine the fediverse, but their accounts get disabled by the local instance admin, and when they try evading that by spinning up a few instances on their own domains to hide their spam or follow bots behind, those instances generally get blocked by instances that see said spam toots or spam followers.
Might be offset by way fewer ads, influencers and platform-driven notifications. I find that those things are the main sources of spam on say Twitter or Instragram.
Federated bayesian filters or local machine learning ?
You could have a personal (stored on device) Bayesian filter that learns from your preferences, and a shared one to block spam spotted by your contacts
As other people suggested, I don't think it should be at the node operator level, for freedom of speech concerns