Skip to main content
dietrich

Firefox Startup Performance Weekly Summary

This week brings a boost in visibility of results, not just for startup, but for all the performance testing we're doing on all branches and platforms. As I mentioned last week, I was working on an automated method of generating the cross-branch startup results. Luckily Johnath and Chris Atlee had done the hard work when making the performance dashboard. It has a JSON file that contains a roll-up of the previous 7 days performance data, which is updated every 5 minutes with new Talos results. Using that as the datasource, I wrote a script summarizes the results for each test+branch+OS combination across all the boxes that returned results, as well as stable branch and previous week differences. The final product gives a snapshot view of how each branch compares to the stable branch.

This is useful for a few different reasons. First, we (and the press and our users and our managers and everyone really) are able to know at-a-glance how any branch compares to the stable release branch. An example of another use is that before the JS team does a Tracemonkey merge, they can quickly see if any major performance effects can be expected.

The table shows red or green for any differences that are outside of a 2% threshold, to take test noise into account. This is quite liberal, as I've already calculated the numbers to take into account whether the difference is within the standard deviation. Reducing the noise in the tests would be a big help - perhaps some researcher will take up Roc's challenge. A second point of trust is my math :) I'm not a statistician, so please view source and let me know where I've miscalculated.

An example of the full report is here. I'm going to file a bug and work with release-engineering to get it pushed out to the graph server, where the dashboard lives. The data from this week's startup table is copied below (though sans the color-coding of the real thing. Actually, now Wordpress keeps stripping out my styles, so you get an ugly table this week).

# Ts

Firefox3.5 Firefox3.6 Firefox TraceMonkey
Leopard median: 1469deviation: 110mean: 1481

from last week: 2%

median: 1115 deviation: 59 mean: 1096 from last week: 3\% from 3.5: 26\% median: 1036 deviation: 48 mean: 1006 from last week: 1\% from 3.5: 32\% median: 1014 deviation: 38 mean: 1000 from last week: 0\% from 3.5: 32\%
Linux median: 625 deviation: 7 mean: 626 from last week: 0\% median: 632 deviation: 7 mean: 633 from last week: 0\% from 3.5: -1\% median: 619 deviation: 10 mean: 623 from last week: 1\% median: 628 deviation: 7 mean: 628 from last week: 0\%
Vista median: 538 deviation: 8 mean: 540 from last week: 0\% median: 533 deviation: 13 mean: 537 from last week: 1\% from 3.5: 1\% median: 503 deviation: 23 mean: 509 from last week: 0\% from 3.5: 6\% median: 511 deviation: 41 mean: 531 from last week: 11\% from 3.5: 2\%
XP median: 461 deviation: 6 mean: 461 from last week: 0\% median: 464 deviation: 6 mean: 464 from last week: 1\% from 3.5: -1\% median: 448 deviation: 7 mean: 448 from last week: 0\% from 3.5: 3\% median: 530 deviation: 38 mean: 501 from last week: 10\% from 3.5: -9\%
This week's activity:

UPDATE: Fixed link to full report.