Velocity 09: Fistful of Sand: Monitoring Code Performance at MySpace.com

Posted in Conferences, Development, Web Technologies on June 28, 2009



Every good developer loves building a bonsai-site—a fast, lean paean to minimalism and efficiency. The DOM tree is carefully pruned, lovingly decorated with tasteful CSS. The browser welcomes it with a sigh of relief. The server’s CPU barely registers, and the database brushes delicately against the SAN, no lookup un-clustered.

Inevitably, such perfection is short-lived. Once the site is deployed, the snowball effect begins. A new panel is added on the top left, resulting in an database call. A new button is prominently displayed on the top navigation, resulting in increased traffic to a very heavy report-style page. A fade-in effect is added to the splash page, resulting in a beefy Javascript download for end users. Like a snowball rolling downhill, the site acrues performance-undermining features. MySpace.com is a several-year-old snow boulder and grows faster the larger it gets. In this talk, we will go over a particular performance monitoring tool we have built. The talk will be liberally dosed with ego-deflating vignettes illustrating our periodic and spectacular mishaps.

System Overview

Over the years, we have witnessed many performance-related disasters at MySpace. These typically resulted in much running around, yelling over red-lined graphs, execution of hastily written scripts, sloppy downings of overheated coffee, and, in the end, the inevitable question: “How can this be prevented in the future?” Usually the answer to this question was, “if only we had avoided this database call.” “If only we had cached this lookup table.” “If only we had avoided this redundant lock.”

The road to performance Hell is paved with “if only”s.

The goal of the system we built is to provide developers with useful performance-related information before their code reaches our staging and deployment process. The system mixes a few hard and fast rules (“avoid inline script tags”) with analytics (“your page allocated 120kb of memory for the last ten months—after your latest commit it allocated 500kb”) to give the developer a sense of what has changed.

Every night the site is compiled and deployed to a series of testing servers. This process is totally automated and requires no developer intervention. The profiling consists of two main parts. Client side profiling is used to gauge the performance of the HTML payload in the browser. Server side profiling is used to profile how an individual web server will interact with the MySpace ecosystem.

Client Side Profiling

The Client-side profiling portion of our system tracks everything that happens from the moment the HTML leaves our web server to the moment the page is completely rendered on the user’s browser.

This system is a combination of 2 components; an Internet Explorer Plugin that tracks the performance and HTML events on the user’s machine, and a custom HTTP Proxy which simulates a slow connection and injects custom javascript code into the tested page.

Using this system, we track and analyze the following data points:

  • Render Time Profiling o Time it took for each HTML element to be rendered on the browser o Which element takes the longest to be rendered o Automatic identification of rendering bottlenecks o Which HTML/JavaScript code was executed during rendering of each element o Which files were downloaded during the rendering of each element
  • Downloaded Data o Total download size o Size of external stylesheets o Size of external Javascript o Size of linked images
  • Performance Data o Processor Time – CPU Usage on the client side (Browser’s process) o Private Working Set – Memory state of the browser during page rendering
  • Source Validation o A complete list of HTML guidelines/rules which we want to avoid; such as “Don’t put <link> tags in the HTML’s “<body>“

Server Side Profiling

The Server Side Profiler runs at the same time as the Client Side Profiler, tracing the execution and operating system interaction of the server code as the page is rendered. Below are examples of the data point tracked:

  • Number of database calls
  • Number of cache calls
  • Number of external http requests
  • Total number of method calls
  • Redirects
  • Memory allocations by calls and memory totals.
  • Number of locks per request

Developer Tool Bar

The “Developer Tool Bar” is a browser toolbar that gives the developer a complete real-time analisys of his front-end code performance. Our toolbar provides a duplicate method of testing the HTML payload before a developer checks in their source. This places the importance of performance as far forward as possible in the development cycle.

With this toolbar, our developers could:

  • Profile the rendering of the page under various connection speed
  • Easily identify rendering bottlenecks
  • Test the size of the page’s elements
  • View the load order of each element
  • Take a snapshot of the browser’s CPU and memory footprint
  • Validate the output HTML against our coding standards

Notification System*

The Notification portion of the system allows developers who integrate with the Performance system to notify the subscribed users based on cutomizable criteria.

Examples:

  • Server Side Performance Alerts o Number of Database calls increase o Number of Cache call increase o Amount of memory allocated per requests increases
  • Client Side Performance Alerts o Download size increases o Browser memory working set increases o Average render time increases o Source Validation failures

Chris Bissell

MySpace

Chris is the Chief Software Architect at MySpace.com. In other words, he comes up with overly convoluted ‘design patterns’ in order to lengthen the development process.

Jeremy Custenborder

MySpace.com

Jeremy Custenborder has spent several years in the trenches at MySpace, resolving many bottlenecks and issues.

Yadid Ramot

MySpace.com

Yadid has spent much time mentoring the MySpace development community on client site performance.

Watch Video Watch Video on External Site

Tags: Conferences, Javascript, MySpace, Scalability, HTML, Profiling, Monitoring, Performance, Development, oreilly media, Velocity 2009