Few months ago I’ve switched one of our internal projects from doing synchronous database saves of analytics data to an asynchronous processing using starling + a pool of workers. This was the day when I really understood the power of specialized queue servers. I was using database (mostly, MySQL) for this kind of tasks for years and sometimes (especially under a highly concurrent load) it worked not so fast… Few times I worked with some queue servers, but those were either some small tasks or I didn’t have a time to really get the idea, that specialized queue servers were created just to do these tasks quickly and efficiently.

All this time (few months now) I was using starling noticed really bad thing in how it works: if workers die (really die, or lock on something for a long time, or just start lagging) …

Since the day one when I joined Scribd, I was thinking about the fact that 90+% of our traffic is going to the document view pages, which is a single action in our documents controller. I was wondering how could we improve this action responsiveness and make our users happier.

Few times I was creating a git branches and hacking this action trying to implement some sort of page-level caching to make things faster. But all the time results weren’t as good as I’d like them to be. So, branches were sitting there and waiting for a better idea.

Few months ago my good friend has joined Scribd and we’ve started thinking on this problem together. As the result of our brainstorming we’ve managed to figure out what were the problems preventing us from doing efficient caching: …

Catchy title? Its a webminar hosted by Robert Scoble, with panel members like Matt Mullenweg (WordPress - their extensive use of PHP, MySQL and more, and scalable even for wordpress.com), Paul Bucheit (FriendFeed, creator of GMail) and Nat Brown (iLike, a pretty popular Facebook application), you’d be silly not to miss it.

Its all about building a scalable server environment that grows with your traffic (virtually overnight, in some cases). I hope its all fairly generic and not Rackspace specific… we should learn to have these “fun” panel webminars.

Seriously kids, what’s with the lack of scalability? I’ve never seen CNN or the NYTimes go down on “trimmed” versions.

Is it a question of bandwidth? Is it lack of hardware?

Take for example, Malaysiakini (the first alternative news source in Malaysia, with a subscription model built around it). It runs FreeBSD, uses PostgreSQL, and has a CMS on top of it (so almost a LAMP stack right there). There’s even use of Squid for caching. Yet there’s lacking load balancing? This is where the cloud can come into play, when there’s high traffic.

Next up, …

So, I wrote about the begining of our wild database issues. Since then, I have been fighting a cold, coaching little league football and trying to help out in getting our backup solutions working in top shape.  That does not leave much time for blogging.

Never again will we have ONLY a cold backup of anything.  We were moving nightly full database dumps and hourly backups of critical tables over to that box all day long.  Well, when the filesystem fails on both the primary database server and your cold backup server, you question everything.  A day after my marathon drive to fix the backup server and get it up and running, the backup mysql server died again with RAID errors.  I guess that was the problem all along.  In the end, we had to have a whole new RAID subsystem in our backup database server.  So, my …

Hot on the heels of news that SquareSpace is using Oracle, comes news that LinkedIn is going whole hog with MySQL.

Actually, you could say that LinkedIn is buying into Sun. They are buying the MySQL Enterprise subscription and they'll be running MySQL on Sparc servers and Solaris 10. They've signed up for Sun Professional Services, MySQL Professional Services, and …

Today I was doing some work on one of our database servers (each of them has 4 SAS disks in RAID10 on an Adaptec controller) and it required huge multi-thread I/O-bound read load. Basically it was a set of parallel full-scan reads from a 300Gb compressed innodb table (yes, we use innodb plugin). Looking at the iostat I saw pretty expected results: 90-100% disk utilization and lots of read operations per second. Then I decided to play around with linux I/O schedulers and try to increase disk subsystem throughput. Here are the results:

Tag1 Consulting is focused on improving Drupal's performance and scalability. We also believe that when information is freely shared, everyone wins. Toward these ends, we are working on an online book titled, "Drupal Performance and Scalability". The book is divided into five main sections, Drupal Performance, Front End Performance, Improved Caching and Searching, Optimizing the Database Layer, and Drupal In The Cloud. The book is primarily aimed toward users running …

As I said before, I was invited to be on a panel at Velocity Conference.  I was delighted to go.  I had never been to San Francisco.  I have been to Portland and Santa Clara several times.  The panel was great.  It was the Brian and photo sharing sites show.  Seriously, it was me (dealnews.com), John Allspaw of Flickr, Don MacAskill of SmugMug and Farhan Mashraqi of Fotolog.  Oh, there was also Shayan Zadeh of Zoosk, a social dating network and Michael Halligan, a consultant from BitPusher.  We all had similar ideas.  I told my …

Michael Arrington of TechCrunch asks Twitter a few questions. I have only included a sample list below but you should read his blog post for all the questions:

• Is it true that you only have a single master MySQL server running replication to two slaves, and the architecture doesn’t auto-switch to a hot backup when the master goes down?
• Do you really have a grand total of three physical database machines that are POWERING ALL OF TWITTER?
• Is it true that the only way you can keep Twitter alive is to have somebody sit there and watch it constantly, and then manually switch databases over and re-build when one of the slaves fail?
  

A 'yes' answer to any of these questions by Twitter would be disturbing to say the least. However, …