Ken,

With all due respect, I've asked you and Heikki to look into these issues for *4 years*, and MySQL to do the same, with the same end-result: no movement.

You absolutely deserve props for the AUTO_INCREMENT patch. IMHO, it took too long to arrive, but better late than never - it works very well, no doubt. And I'm positive the InnoDB plugin was a big win for lots of your customers, but it wasn't for us. I prefer on-disk compression using ZFS to table compression, and disk spaces / throughput aren't our bottlenecks: we have more storage and IOPS capacity than InnoDB is able to make use of.

Clearly, you guys have been working hard on what you think is important for your customers. Which means, just as clearly, we must not be your target customer. There's nothing wrong with that - I respect businesses that choose their customers carefully and then do everything to support them. That's how I run my own business.

I just wish I could convince you that I'm a good target customer, but after years of pain and pleading, it's just become apparent that just isn't going to happen. So we've had to find solutions elsewhere. Percona stepped up to the plate.

You're absolutely right about the nature of community patches, and it's something that keeps me up at night worrying, but at the same time, InnoDB right now is so hard to scale, I'm stuck between a rock and a hard place: A site that's down or community patches that might affect our workload in unknown ways.

Given those two choices, I'd rather be part of the solution: by running our workload on the community patches, and providing feedback, we can help work out the kinks in those patches. It might make for some short-term pain, but we believe that it'll result in long-term gains. We're ok with that. *knocks on wood*

Of course, we roll these things out gradually. We set up one slave out of production, test on it, then gradually move it into production, test on it, then roll it out to more slaves and clusters. It's not like we're just replacing every MySQL instance overnight with these patches - we're trying to carefully gauge their impact as best we can.

Honestly, it's sad we have to go down this route. I'd much rather pay Oracle or Sun or whomever will take my money to solve these problems for us instead of going the community route. But what choice do I have?

MySQL or Sun could fork InnoDB. This seems inevitable, but it's taking them awhile to come to that conclusion.

Oh, and comments seem to be broken on your blog - the CAPTCHA wasn't working for me, so I coudln't reply to your post the other night.

Personally, I'm hoping Drizzle solves that for us. Like I said, I'm hoping Drizzle + XtraDB gets us on the right path.

If not, you're absolutely right, MySQL is going to be a problem. :(

Thanks again for all of your work on these issues. I'm not sure if you hadn't "broken the ice" and released all those great patches that we'd be where we are today. Your patches were the catalyst for all of this InnoDB forward motion this year, I think. If there's ever anything I can do to repay that effort, I'm all ears - just let me know.

Thanks!

I'm not sure I understand your assertion. Are you saying, for example, that Facebook shouldn't have more than "a few dozen transactions running simultaneously" across their entire platform? How are they supposed to accomplish that with hundreds of millions of users? And if they do, they have poor design and/or query optimization?

That doesn't seem to make sense, so I must be misunderstanding your point...

We bought 4 clustered configs, yes. S7410c I think they're called. Two head units, each with one "Readzilla" SSD read accelerator, and one JBOD with 22 1TB disks and 2 "Logzilla" SSD write accelerators.

So there are an awful lot of things wrong with this comment, but chief among them is the assumption that SmugMug's growth has somehow flattened or is in decline. That couldn't be further from the truth - we had explosive growth this year, despite our expectations of slowing thanks to the economic meltdown.

The other element you're overlooking is that when Amazon lowers their S3 pricing (like they did this year), *all* of our data storage costs get reduced, not just the "new" stuff. That's a very different model than capitalizing our own storage - those costs are then not only sunk, but fixed.

Further, capitalizing those storage costs requires cash. Lots of it. It's easy to say you'll just amortize it over 5 years, but you still have to come up with the cash up front or carry a nasty debt load. (Look where the debt load in this country got us this year). We like owning our business, being our own masters, and not owing anyone - investors or creditors - a dime. Amazon's cost structure allows us to do that - and personally, I'm not sure I can put a price on that sort of freedom.

Finally, Amazon has serious incentive to keep lowering prices and keeping us happy. The "cost of moving" from Amazon is roughly equal to one month's "rent" so there's very little barrier to simply switching to something else (in-house storage, Windows Azure, Google AppEngine, etc). That keeps them honest, and keeps our operating cost structure down. The same cannot be said of doing your own storage in-house. Once you own that stuff, you have to monetize it.

So yes, we're still extremely happy with Amazon, and extremely happy with the less-tangible benefits that come along with it (lower headcount, no venture capital, no debt, flexibility, less mental anguish, etc) along with the obvious business savings and optimizations.