There are surely a lot of very smart people there, but the ones I worked with had all drunk the kool-ade that Raid 10 on 15K spindles with ZFS was so awesome that there couldn't be IO problems.
When you have a company that is using technology that is essentially beta (like ZFS) and you have to pay large amounts of money to Sun every time you need support, that creates an incentive to pretend the problem doesn't exist. Joyent also showed some signs of serious cash flow issues (half off if you sign up for a year) which may explain why they couldn't pay Sun enough money to identify and/or address the issues.
Note: Switching the app that I mentioned to a $100/month slice (off of a $400/month zone) resulted in a massive speedup and noticeably faster disk IO, db performance, etc.
When you have a company that is using technology that is essentially beta (like ZFS) and you have to pay large amounts of money to Sun every time you need support, that creates an incentive to pretend the problem doesn't exist. Joyent also showed some signs of serious cash flow issues (half off if you sign up for a year) which may explain why they couldn't pay Sun enough money to identify and/or address the issues.
Note: Switching the app that I mentioned to a $100/month slice (off of a $400/month zone) resulted in a massive speedup and noticeably faster disk IO, db performance, etc.