METRICS.md

Firefox Screenshots Metrics

Last Update: 2017-11-06

This document is a summary of the metrics Firefox Screenshots will record, how we're recording them, and what we're looking for in those metrics. There are two main areas we'll look at:

0. Metrics to tell us if Firefox Screenshots is affecting Firefox. For instance, do Firefox Screenshots users engage longer with Firefox?

1. Help us understand how people use Firefox Screenshots, so that we can improve the experience of the tool itself. A/B tests fall under this umbrella.

Additionally, it's important to recognize that Firefox Screenshots is an add-on in a browser as well as a server-side service.

accountId / deviceId / user ID

Each device that takes Screenshots (browser, profile, computer) generates a random UUID ID. We call this the deviceId. In addition a random secret is generated. The two together are used to register and authenticate with the server.

The deviceId is only registered with the screenshots.firefox.com server when the user first saves a shot. Starting Screenshots or downloading does not cause registration.

An alternate way to authenticate with server is by successful signin to firefox account. An accountId which is a uid returned by fxa-auth-server is assigned once a user login with firefox account and its value is reset on signout from settings.

Each event is sent to the server with its accountId or deviceId. The server hashes this ID (combining it with a random server string that is rotated occasionally) before sending it to Google Analytics. This hashed ID is generally called cid (customer ID, a Google Analytics term).

For web page views (viewing and interacting with individual shots or My Shots) we serve up /ga-activation.js or /ga-activation-hashed.js (the latter for pages with private URLs). This file is generated per user and includes the cid, if the user is authenticated with the service.

The deviceId does not change for the life of a browser profile.

Key Metrics

Key metrics of Firefox Screenshots are fairly simple:

Do Screenshots users tend to use Firefox more?

Unlike the rest of the metrics, answering this question requires using Mozilla's Telemetry system, which tracks usage for Firefox as a whole. In order to compare engagement between Firefox sessions that do and don't have Screenshots usage, three scalars were added to the main Telemetry ping:

• screenshots.download - incremented each time a shot is created & downloaded during a session
• screenshots.upload - incremented each time a shot is created & uploaded to the screenshots server during a session
• screenshots.copy - incremented each time a shot is created & copied straight to the clipboard without downloading or uploading

Do people continue to create shots?

This will be tracked in a cohort graph.

Do people share those shots with other people?

In this case there is not a "right" answer, but sharing indicates a different kind of use case from personal storage.

We can look at the share event in GA, and we can look at non-owner visits (web/visit/non-owner vs web/visit/owner). We also have some referrer access via GA. These reports should be accessible within GA given the metrics listed below.

Do people revisit their own shots?

This is primarily a count of web/visit/owner. The tool always opens the page once, and will fire web/visit/owner-first on that first visit.

Do people who view a shot, install Firefox Screenshots?

We will be tracking some events under goto-pageshot that would lead people from shot pages to a place where they could load the page. Then we track clicking the install link itself, which GA reporting should be able to connect to the original goto-pageshot event. We can't detect how often that install click leads to an actual install.

Data Collection

Firefox Screenshots assigns each user a random ID (associated with their profile) when the add-on is installed. This ID is associated with all shots the user makes. For the purpose of Google Analytics (GA) the ID is hashed. The same hashed ID is used for website visits and events, and for add-on events.

No metrics data is sent to any servers until a user interacts with Firefox Screenshots.

The add-on does not communicate directly with GA, instead it POSTs an event to the Firefox Screenshots server which, in turn, sends the event to GA. This model allows us to only send the data we need for analysis instead of the more comprehensive data collection that google-analytics.js does. Besides the event information specified below we also send the User-Agent string, and the add-on provides:

• Any A/B tests the user is in, or if the user is in the control for a test
• The application (“firefox”)
• The add-on version (“6.0.0”)
• For selections we send the size of the selection, rounded to 10 pixels. E.g., 500 x 150

Metrics schema

Each item in these events requires:

Event category: maps to the "source": addon or web

Event action: what the event "does", such as start-shot (note that Save actually "takes" the shot, the focus should be on what happens as a result of interacting with the control)

Event label: exactly what control invoked the action, such as toolbar-button. These are the "locations":

• toolbar: the browser toolbar
• topbar: the top bar on the page during selection
• overlay: the question overlay after you click the toolbar button
• navbar: the top navigation bar on a shot page
• content: the non-navigation-bar parts of a shot page
• myshots: the shot index page
• selection: anything that happens during the selection phase, that doesn't happen in the topbar
• keyboard: any keyboard shortcut used

Login status custom dimension

The cd9 dimension holds the "login type". The values can be:

• non-login: visitor is not logged in
• extension: logged in via the extension, with no FxA login
• extension-fxa: logged in via the extension, With a FxA login
• browser-fxa: using a non-Firefox browser, with a FxA login

A/B tests

Test alternate share panel

As described in #3620, we are showing some users a different share icon on the My Shots and single shot pages. The test is named shotShareIcon.

The dimension cd7 will be control (Photon design system icon) or newicon (an Android-style "less than"-style icon, 10% of population).

(finished) Highlight shot button

As described in #2081 we are putting a badge on the button of some users. The test is named highlightButtonOnInstall.

The dimension cd3 will be control (no badge) or badge (10% of population).

If the badge is shown we send the event addon/ab-highlight-button-shown – this is marked as a non-interactive event.

(finished) Adjust My Shots button display

As described in #2082 we are changing how the My Shots button is displayed. The test is named styleMyShotsButton.

The dimension cd4 will be control (normal styling) or something else Yet To Be Determined.

(finished) Auto-open share panel

As described in #2079 we auto-open the share panel for a set of users. The test is named autoOpenSharePanel.

The dimension cd5 for the creator will be control (no auto-opening) or autoopen (share panel will be auto-opened). The dimension cd6 will be control (no auto-opening happened) or autoopen (auto-open happened in the past at shot creation).

The primary change was in server/src/pages/shot/share-buttons.js

Add-on metrics

1. Toggle shot button on addon/start-shot/toolbar-button (previous to 54 launch the label was toolbar-pageshot-button)
2. Use keyboard shortcut to start shot addon/start-shot/keyboard-shortcut (accel-alt-control-c) (FIXME: not yet implemented)
3. Use the right-click context menu to start a shot addon/start-shot/context-menu
4. Start shot with onboarding because the site requested it (typically a visit to /#hello) addon/start-shot/site-request
5. Make a selection addon/make-selection/selection-drag with cd2: {px width}, cd1: {px height}
6. Make a selection by clicking on an element addon/make-selection/selection-click with cd2: {px width}, cd1: {px height}
7. Click but fail to find an element that can be selected addon/no-selection/no-element-found (error case, not sure when it happens)
8. Cancel a selection by clicking on the background addon/cancel-selection/selection-background-mousedown
9. Start resizing the selection addon/start-resize-selection/handle
10. Finish resizing the selection addon/resize-selection/mouseup
11. Finish resizing, without a change addon/keep-resize-selection/mouseup (mousedown, don't move, mouseup)
12. Start moving the selection addon/start-move-selection/selection
13. Finish moving the selection addon/move-selection/mouseup
14. Finish moving, without a change addon/keep-move-selection/mouseup
15. Click Save addon/save-shot/overlay-save-button
16. Click Cancel addon/cancel-shot/overlay-cancel-button
17. Click Download addon/download-shot/overlay-download-button
18. Click Copy addon/copy-shot/overlay-copy-button
19. Copy to clipboard keyboard shortcut addon/copy-shot/keyboard-copy
20. Cancel because URL changed addon/cancel-shot/url-changed (when something that uses window.history "navigates" spontaneously away)
21. Cancel because the tab is navigated (such as entering something in the URL bar), or the tab was closed, or the tab was reloaded addon/cancel-shot/tab-load (previously closing would emit addon/cancel-shot/tab-close and addon/cancel-shot/tab-reload for those cases)
22. Click My Shots addon/goto-myshots/selection-button
23. Go to My Shots by hitting the Screenshots button on a about:newtab page addon/goto-myshots/about-newtab
24. Go to /#hello to onboard (because the user pressed the Screenshots button on a page that could not be shot) addon/goto-onboarding/selection-button
25. Click on "Save visible" addon/capture-visible/selection-button
26. Click on "Save Full Page" addon/capture-full-page/selection-button
27. Click Cancel after previewing full-page/visible addon/cancel-shot/cancel-preview-button
28. Click Save after previewing full-page shot addon/save-full-page/save-preview-button
29. Click Save after previewing visible shot addon/save-visible/save-preview-button
30. Click Download after previewing full-page shot addon/download-full-page/download-preview-button
31. Click Download after previewing visible shot addon/download-visible/download-preview-button
32. Click My Shots button from error panel addon/goto-myshots/error-panel
33. Hit Escape (Cancel) addon/cancel-shot/keyboard-escape
34. Hit Enter (Save) addon/save-shot/keyboard-enter
35. Encounter an error saving the shot addon/error/save-shot
36. Encounter any other kind of error addon/error/misc
37. Install the add-on addon/install (fired internally, regardless of how it is installed)
38. Uninstall the add-on addon/uninstall (fired internally, regardless of how it is uninstalled)
39. Hit shot button on a page that can't be shot (XUL page) addon/abort-start-shot/xul-page
40. Hit shot button on a page that uses <frameset> and can't be shot, addon/abort-start-shot/frame-page
41. Hit shot button on a page (like an SVG image) that doesn't have a document.body and can't be shot, addon/abort-start-shot/document-is-TAGNAME where TAGNAME is the tag of document.documentElement (e.g., document-is-svg)
42. Toggle shot button off addon/cancel-shot/toolbar-button
43. Bad response when trying to login addon/login-failed/bad-response-CODE
44. Connection error trying to login addon/login-failed/connection-error
45. Bad response when trying to register addon/register-failed/bad-response-CODE
46. Connection error trying to register addon/register-failed/connection-error

Deprecated:

1. Hit shot on any about: page, addon/start-shot-about-page (this has been disallowed)
2. Hit shot on any non-http page, addon/start-shot-non-http/actual-scheme (only file: would be possible, and we no longer track)
3. Test pilot was present at install time addon/test-pilot-installed~~
4. Test pilot was not present at install time addon/test-pilot-not-installed~~
5. Closing a tab would emit addon/cancel-shot/tab-close, now it emits addon/cancel-shot/tab-load
6. Reloading a tab would emit addon/cancel-shot/tab-reload, now it emits addon/cancel-shot/tab-load
7. Previously to the 54 launch there was addon/start-shot/toolbar-pageshot-button which was renamed to addon/start-shot/toolbar-button

Onboarding metrics

The onboarding slides have some events:

1. Click on the next-slide button: addon/navigate-slide/next
2. Click on the prev-slide button: addon/navigate-slide/prev
3. Navigate to the next slide by hitting ArrowRight: addon/navigate-slide/keyboard-arrowright
4. Navigate to the previous slide by hitting ArrowLeft: addon/navigate-slide/keyboard-arrowleft
5. Click on one of the dots to navigate to a specific slide: addon/navigate-slide/goto
6. Cancel the slides by clicking on the background: addon/cancel-slides/background-click
7. Cancel the slides by clicking on skip: addon/cancel-slides/skip
8. Cancel the slides by hitting Escape: addon/cancel-slides/keyboard-escape
9. Finish the slides by clicking on done: addon/finish-slides/done
10. Visit a slide through any kind of navigation: addon/visited-slide/slide-INDEX (1-4)

Internal add-on events

1. Start an upload addon/upload/started with eventValue of Kb (1000 bytes)
2. Fail to upload due to connection aborted addon/upload-failed/connection (up to version 1.0.1 was addon/upload/failed-connection)
3. Fail to upload due to bad status from server addon/upload-failed/status-{code} (up to version 1.0.1 was addon/upload/failed-status with eventValue: status code)
4. Finish upload successfully addon/upload/success
5. After failure, re-attempt the upload addon/upload-retry/times-{N} (up to version 1.0.1 was addon/upload/upload-retry with eventValue: times (1-2)) (FIXME: we have no retry) sendEvent("click-install-firefox-home", {useBeacon: true});

Add-on performance measurements

Performance measurements use the GA User Timings API, instead of the regular Event API.

User Timings schema

Each item in these events requires:

Timing category: maps to the "source": addon or web

Timing action: what kind of performance measure, currently just perf-response-time, which measures the "response" in the RAIL performance model: the time from a user interaction (like a button click) to a user-visible change in the UI.

Timing variable: which event's performance is being measured. This generally is the same as the name of the event used to start the measurement, such as start-shot.

Timing value: the number of milliseconds associated with the variable. For perf-response-time, the response time in milliseconds.

Internal-only events

Internal-only events are used to help measure user timings, but aren't useful to record on their own.

NOTE: Internal-only events are not submitted to GA.

1. New tab opened for newly-created shot page addon/internal/open-shot-tab
2. Screenshots UI hidden by uicontrol addon/internal/deactivate
3. Pre-selection iframe shown addon/internal/unhide-preselection-frame
4. Selection iframe shown addon/internal/unhide-selection-frame
5. Preview iframe shown addon/internal/unhide-preview-frame

First step: starting the shot

1. Time from clicking the page action (or toolbar button) to displaying the preselection iframe, addon/perf-response-time/page-action

• Start: addon/start-shot/toolbar-button
• End: addon/internal/unhide-preselection-frame

1. Time from clicking the context menu item to displaying the preselection iframe, addon/perf-response-time/context-menu

• Start: addon/start-shot/context-menu
• End: addon/internal/unhide-preselection-frame

Second step: choosing the shot contents

1. Time from initiating a selection on screen to seeing the selection, addon/perf-response-time/make-selection

• Start: addon/make-selection
• End: addon/internal/unhide-selection-frame

1. Time from clicking the 'full page' button to displaying the preview iframe, addon/perf-response-time/capture-full-page

• Start: addon/capture-full-page
• End: addon/internal/unhide-preview-frame

1. Time from clicking the 'save visible' button to displaying the preview iframe, addon/perf-response-time/capture-visible

• Start: addon/capture-visible
• End: addon/internal/unhide-preview-frame

Third step: upload or download

For uploads, the measurement is from clicking the save button to a new tab being opened:

1. Save a selection shot (Enter key or button click), addon/perf-response-time/save-shot

• Start: addon/save-shot
• End: addon/internal/open-shot-tab

1. Save a full page shot, addon/perf-response-time/save-full-page

• Start: addon/save-full-page
• End: addon/internal/open-shot-tab

1. Save a truncated full page shot, addon/perf-response-time/save-full-page-truncated

• Start: addon/save-full-page-truncated
• End: addon/internal/open-shot-tab

1. Save a visible selection shot, addon/perf-response-time/save-visible

• Start: addon/save-visible
• End: addon/internal/open-shot-tab

For downloads, because Firefox doesn't always show download UI, the measurement is from clicking the download button to the screenshots UI being hidden:

1. Download a selection shot, addon/perf-response-time/download-shot

• Start: addon/download-shot
• End: addon/internal/deactivate

1. Download a full page shot, addon/perf-response-time/download-full-page

• Start: addon/download-full-page
• End: addon/internal/deactivate

1. Download a truncated full page shot, addon/perf-response-time/download-full-page-truncated

• Start: addon/download-full-page-truncated
• End: addon/internal/deactivate

1. Download a visible selection shot, addon/perf-response-time/download-visible

• Start: addon/download-visible
• End: addon/internal/deactivate

Owner web visit

These are events that an add-on user can encounter on a shot they own

1. Visit the page, web/visit/owner
2. Visit the page immediately after it is created (as part of the normal flow), web/visit/owner-first
3. Click delete web/start-delete/navbar
4. Confirm delete web/delete/popup-confirm
5. Cancel delete web/cancel-delete/popup-confirm
6. Click delete from shot index, web/start-delete/my-shots
7. Confirm delete from shot index web/delete/my-shots-popup-confirm
8. Cancel delete from shot index, web/cancel-delete/my-shots-popup-confirm
9. Click My Shots web/goto-myshots/navbar
10. View expired shot web/view-expired/owner
11. Recover expired shot web/recover-expired
12. Visit original page web/view-original/navbar-owner
13. Visit original page from expired view web/view-original/expired-owner
14. Click share web/start-share-owner/navbar
15. Click Facebook web/share-owner/facebook
16. Click Twitter web/share-owner/twitter
17. Click Pinterest web/share-owner/pinterest
18. Click mailto web/share-owner/mailto
19. Hit copy web/share/copy
20. Copy with "rich copy", web/share/rich-copy
21. Focus link field web/share/focus-url
22. Cancel/close share web/cancel-share
23. Visit Screenshots link from footer web/goto-homepage/footer (was eventAction goto-pageshot before 54 launch)
24. Click "Copy Image Text" on the context menu web/copy-image-text/context-menu
25. Click Feedback/mailto button start-feedback/footer
26. Click on clip web/goto-clip/content
27. Click on clip in a mobile browser web/mobile-download/content
28. Click the download button web/download/navbar
29. Visit an image directly, when the image isn't embedded directly in a Screenshots shot page, web/visit/direct-view-owner
30. View an image directly, when the image is being shown as part of a Facebook/Twitter style preview (the og:image or twitter:image), web/visit/direct-view-embedded-owner
31. Close new edit tools promotion dialog, web/promo-closed
32. Add shot to favorites web/set-favorite/navbar
33. Remove shot from favorites web-unset-favorite/navbar
34. Signin to Firefox Accounts web/fxa-signin
35. Signin to Firefox Accounts from banner web/fxa-signin-ad-banner
36. Signin to Firefox Accounts from onboarding promo web/fxa-signin-onboarding-promo
37. Click on dismiss of Firefox Accounts onboarding promo web/onboarding-promo-closed

Shot Index (My Shots)

1. Click a tile web/goto-shot/myshots-tile
2. Click on the original link web/goto-original-url/myshots-tile
3. Enter search term and submit web/search/submit
4. Enter search term (at least 4 characters) and wait 1 second web/search/timed
5. Clear search term web/clear-search/submit and click the arrow
6. Clear search term web/clear-search/keyboard by simply backspacing
7. Receive no search results web/search-no-results
8. Right-click (or get the context menu) anywhere on the page contextmenu/background, contextmenu/shot-tile, contextmenu/search, or contextmenu/header depending on where the user clicks.
9. Click download from tile: web/download/myshots-tile
10. Clear search with button: web/clear-search/button
11. Click to favorite: web/favorite/myshots-tile
12. Click to un-favorite: web/unfavorite/myshots-tile

Settings (My Shots)

1. Click connect button web/start-connect/settings
2. Click disconnect button web/start-disconnect/settings
3. Confirm disconnect on settings page web/confirm-disconnect/settings-popup-confirm
4. Cancel disconnect on settings page web/cancel-disconnect/settings-popup-confirm

Non-owner web visit

1. Visit the page, web/visit/non-owner
2. Visit an image directly, when the image isn't embedded directly in a Screenshots shot page, web/visit/direct-view-non-owner
3. View an image directly, when the image is being shown as part of a Facebook/Twitter style preview (the og:image or twitter:image), web/visit/direct-view-embedded-non-owner
4. Click flag button web/start-flag/navbar
5. Click Share (same as for owner, but with share-non-owner instead of share-owner, and start-share-non-owner)
6. Visit original URL web/view-original/navbar-non-owner
7. Click Screenshots link in header web/goto-homepage/navbar (was goto-pageshot eventAction before 54 launch)
8. View expired shot web/view-expired/non-owner
9. Click on clip (already covered)
10. Click on the "Get it here" (install Screenshots) banner: web/click-install-banner
11. Click on the "Get Firefox now" (install Firefox) banner: web/click-install-firefox-shot. Also note the link uses ?utm_source=screenshots.firefox.com&utm_medium=referral&utm_campaign=screenshots-acquisition&utm-content=from-shot on the link.

Home Page Events

1. Click My Shots From Home web/goto-myshots/homepage
2. Click on the "Firefox Free Download" (install Firefox) button from home: web/click-install-firefox-home. Also note the link uses ?utm_source=screenshots.firefox.com&utm_medium=referral&utm_campaign=screenshots-acquisition&utm-content=from-home on the link.

Other web events

1. Start the "Leave Screenshots" process: server/start-leave-service
2. Complete leaving: server/leave-service-completed
3. From Leave Screenshots, click Delete: web/leave-service/leave-button
4. From Leave Screenshots, click Cancel: web/cancel-leave/cancel-link
5. After leaving, click home button web/home-after-leave/home-link

Server events

1. When an expired shot is deleted (about 2 weeks after expiration) it sends server/clean-deleted-shot with an eventValue of the number of shots cleaned at that moment. (By default these are checked every minute.)
2. When a successful request to /api/login happens, we send server/api-login
3. When the user successfully logs in to their firefox account, we send server/fxa-login

Page views

1. Views of shot pages show up as /a-shot/{hash}
2. Views of images directly, /images/{hash}
3. Views of images that came from og/twitter metadata, /images/embedded/{hash}

The hashed page ID ({hash}) is a simple SHA1(path), with no additional randomness or salt added.

Annotation metrics

1. Open annotation tools: web/start-annotations/navbar
2. Save Edited shot: web/save/annotation-toolbar
3. Cancel Annotations: web/cancel/annotation-toolbar
4. Select pen from annotation toolbar: web/pen-select/annotation-toolbar
5. Select highlighter from annotation toolbar: web/highlighter-select/annotation-toolbar
6. Select crop tool from annotation toolbar: web/crop-select/annotation-toolbar
7. Confirm crop after selection: web/confirm-crop/crop-toolbar
8. Cancel crop after selection: web/cancel-crop/crop-toolbar
9. Click on clear tool: web/clear-select/annotation-toolbar
10. Open color picker: web/color-picker-select/annotation-toolbar
11. Select a color from the color board: web/{color}-select/annotation-color-board
12. Use pen tool: web/draw/pen
13. Use highlight tool: web/draw/highlight
14. Undo edit: web/undo/annotation-toolbar
15. Redo edit: web/redo/annotation-toolbar
16. Select text tool from annotation toolbar: web/text-select/annotation-toolbar
17. Confirm text edit: web/confirm-text/text-toolbar
18. Cancel text edit: web/cancel-text/text-toolbar

General Google Analytics information

This is stuff we get from including ga.js on Screenshots pages.

1. Browser type
2. Location
3. Language
4. Referrals
5. Social referral
6. Search terms
7. Demographic guesses about users
8. See if some pages are "visited" particularly often (we don't have access to the actual page, but can identify that some specific page has a large number of visits)

Database Metrics

We can do queries directly on the database of Firefox Screenshots. These might include:

1. Look at cohorts of individuals, reviewing their shot creation patterns
2. How many people make a couple shots? How many make a lot of shots?
3. See if people are making shots of mostly private or mostly public pages
4. Identify the dimensions of shots
5. Compare shot URLs to a domain allowlist, to determine broad categories of sites that are popular to view

Error reporting

In order to detect errors in the field that we haven't encountered in development, we catch and report most unexpected exceptions. These exceptions are sent to a Mozilla-hosted server that aggregates and reports on these errors. The development team monitors these errors.

In most cases if an exception report has been sent you will also see a popup notification of an error in Screenshots. Exceptions use catcher.unhandled or noPopup, and these are typically unusual cases where a popup would intrude on an otherwise recoverable error.

In addition to the exception message and a stack trace, the error information includes information about your browser (User-Agent), platform, the add-on version, and IP address. We use Raven.js to collect the error reports.

If an exception occurs in the context of screenshotting a specific page, we specifically filter out any reference to the URL or domain of that page.

Error reporting in the add-on can be opted out of in the same way as other metrics (using the general Telemetry preference, see below). Error reporting on the website cannot be opted out of.

Error reporting data

The data collected in error reports includes:

• Date of the submission
• Browser version and User-Agent header
• Operating system
• The Screenshots system add-on version
• The exception message
• The stack trace
• Length of time the add-on was activated
• Exception-specific properties (for example, if there is an error connecting to the server, we add the server response code to the exception)

While the URL of the page you are shooting is not part of this information, we additionally filter out the URL in case of some bug in how we collect this information.

Opt-out

The add-on reads the opt-out preference (datareporting.healthreport.uploadEnabled), labelled "Enable Health Report" in preferences under Privacy and Security. If this preference is false, or if there is any issue trying to fetch the preference, then no data is sent.

The website reads the Do Not Track Header (navigator.doNotTrack) and if it is present and set to 1 the website will not send metrics data. We treat error reporting differently, and it is sent regardless of the Do Not Track setting.

There may be some exceptions to the above for debugging or analysis of malfunctions.