Velocity 09: Fistful of Sand: Monitoring Code Performance at MySpace.com
Every good developer loves building a bonsai-site—a fast, lean paean to minimalism and efficiency. The DOM tree is carefully pruned, lovingly decorated with tasteful CSS. The browser welcomes it with a sigh of relief. The server’s CPU barely registers, and the database brushes delicately against the SAN, no lookup un-clustered.
Over the years, we have witnessed many performance-related disasters at MySpace. These typically resulted in much running around, yelling over red-lined graphs, execution of hastily written scripts, sloppy downings of overheated coffee, and, in the end, the inevitable question: “How can this be prevented in the future?” Usually the answer to this question was, “if only we had avoided this database call.” “If only we had cached this lookup table.” “If only we had avoided this redundant lock.”
The road to performance Hell is paved with “if only”s.
The goal of the system we built is to provide developers with useful performance-related information before their code reaches our staging and deployment process. The system mixes a few hard and fast rules (“avoid inline script tags”) with analytics (“your page allocated 120kb of memory for the last ten months—after your latest commit it allocated 500kb”) to give the developer a sense of what has changed.
Every night the site is compiled and deployed to a series of testing servers. This process is totally automated and requires no developer intervention. The profiling consists of two main parts. Client side profiling is used to gauge the performance of the HTML payload in the browser. Server side profiling is used to profile how an individual web server will interact with the MySpace ecosystem.
Client Side Profiling
The Client-side profiling portion of our system tracks everything that happens from the moment the HTML leaves our web server to the moment the page is completely rendered on the user’s browser.
Using this system, we track and analyze the following data points:
- Performance Data o Processor Time – CPU Usage on the client side (Browser’s process) o Private Working Set – Memory state of the browser during page rendering
- Source Validation o A complete list of HTML guidelines/rules which we want to avoid; such as “Don’t put <link> tags in the HTML’s “<body>“
Server Side Profiling
The Server Side Profiler runs at the same time as the Client Side Profiler, tracing the execution and operating system interaction of the server code as the page is rendered. Below are examples of the data point tracked:
- Number of database calls
- Number of cache calls
- Number of external http requests
- Total number of method calls
- Memory allocations by calls and memory totals.
- Number of locks per request
Developer Tool Bar
The “Developer Tool Bar” is a browser toolbar that gives the developer a complete real-time analisys of his front-end code performance. Our toolbar provides a duplicate method of testing the HTML payload before a developer checks in their source. This places the importance of performance as far forward as possible in the development cycle.
With this toolbar, our developers could:
- Profile the rendering of the page under various connection speed
- Easily identify rendering bottlenecks
- Test the size of the page’s elements
- View the load order of each element
- Take a snapshot of the browser’s CPU and memory footprint
- Validate the output HTML against our coding standards
The Notification portion of the system allows developers who integrate with the Performance system to notify the subscribed users based on cutomizable criteria.
- Server Side Performance Alerts o Number of Database calls increase o Number of Cache call increase o Amount of memory allocated per requests increases
- Client Side Performance Alerts o Download size increases o Browser memory working set increases o Average render time increases o Source Validation failures
Chris is the Chief Software Architect at MySpace.com. In other words, he comes up with overly convoluted ‘design patterns’ in order to lengthen the development process.
Jeremy Custenborder has spent several years in the trenches at MySpace, resolving many bottlenecks and issues.
Yadid has spent much time mentoring the MySpace development community on client site performance.