Velocity 09: John Adams, "Fixing Twitter: Improving the Performance and Scalability"
Twitter (twitter.com) is a micro-blogging service that allows users to answer the question “What are you doing?”. It takes a simple 140 character string (a tweet), and distributes that to a user’s following (or “friends”) list.
Over the last few years, Twitter has experienced a variety of challenges scaling its site to serve millions and millions of users. From hosting issues to Ruby on Rails, we’ve met many scaling challenges and would like to share some of our experiences with the community.
We will attempt to cover:
- Best practices for deploying Ruby on Rails in production
- Performance impacts of abuse and why rate limiting is so important.
- What to do when your API use overtakes normal web use
- Asynchronous versus Synchronous processing during request lifetime
- Why disk is the new tape (or, how social media isn’t possible without cheap memory)
- Caching methodologies and Twitter’s open source efforts (Scarling, CacheMoney, Libmemcached)
- Why databases are not the best solution for all problems.
- Message queues
- Handling large logs with Thrift
- Incremental scaling improvements: Find the weak point, fixing it, find the next one, repeat.
John Adams (Twitter Operations) has worked in web security, operations, and systems engineering for over 15 years. Prior to Twitter, he has worked at Apple, Inktomi, c|net, and a major video-on-demand site, improving security, performance, and reliability at large scale.