I'd say that even if the task is "a compiler for an existing language" the empirical results are far from stellar, and indeed, border the terrain of comedy.
Claude has made my Ruby compiler pass 8k more rubyspecs tests over the last two weeks, and that includes solving crashing bugs in hours that I'd failed to fix for weeks in the past.
Compilers for existing languages is if anything one of the lower bars for LLMs, given existing implementations or test suites provides an oracle to test against.
It's not lack of ability that is stopping this, but that it's a space where very few people are experimenting and willing to burn enough tokens.
