Wikimedia monthly activities meetings/Quarterly reviews/Wikipedia Zero/June 2013

The following are notes from the Quarterly Review meeting with the Wikipedia Zero team on June 28, 2013

Present: Kul Wadhwa, Dan Foy, Amit Kapoor, Adele Vrana, Adam Baso, Tomasz Finc, Frank Schulenburg, Erik Möller, Sue Gardner, Tilman Bayer (taking minutes)

Participating remotely: Ken Snider, Yuri Astrakhan

Proposed Agenda:

• Partnership Update
• Analytics/Stats
• Tech update
• Feedback from Africa
• USSD pilot update
• Product review
• Needed improvements
• Goals for next quarter

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Kul: welcome
Everyone: Introductions
Kul: recalls February review
agenda today: last few months - review partnerships, stats
big area going forward: awareness building
finally added two fulltime engineers. last time we were 29 deployments behind. we focused on infrastructure changes
(also today:) WP via text
Africa; new research
open issues
goals

12 countries last quarter, now 15
biggest increase in program so far
just 3 additional countries, but tripled footprint in terms of people reached - focused on launches this quarter
Pakistan and Russia launches were biggest so far
Erik: clarifies that launch means WP0 actually becoming available to users, as opposed to start of partnership
(Kul:) USSD/SMS: as discussed last time, we want this to be free - difficult discussion, but moving
Frank: how reliable are numbers on overall program growth now?
Amit: fixed problems (Opera), should be reliable now; have automatic alerts in place now if e.g. partners' IPs change
Sue: possible that people try WP0 and then drop it?
Amit: possible, yes
higher yield from partners: page views per 1 million customers went up 68% from February to May
big impact in smaller countries: 5 countries now (our oldest launches, about a year or more) where 33% of all mobile WP views are now free
Frank: this is what we told the Board in January: we need some patience
Amit: example Botswana; now more free mobile views than total mobile views a year ago
automated dashboard
total view of program http://gp.wmflabs.org/graphs/free_mobile_traffic_by_version_monthly
e.g. compare partner's growth to rest of country
will add breakdown by language version of Wikipedia
big marketing campaigns:
Kul: Amit worked with partners giving advice on campaigns, although we don't get a say on final
e.g. Russian campaign this week; 50 million impressions in 2 months (would be $75k or so spend)
Pakistan: posters near campuses, online ads on news websites
"Dialog" launch this week: screenshots, radio ads, airtime scratch cards, online ads
Kul: we could spend a lot of time investigating what works best, recommend to partners

Dan:
now have two full-time engineers, Adam and Yuri

• worked off the majority of our critical issues backlog
• fixed "leaks" where data charge warning was missing
• now can do live test without bothering Ops
• put information for partners in one place
  

shows bug example (on Nokia with Symbian CSS - not what we typically used in testing)
Kul: tested in Kenya on the ground ourselves, or asked community members to help, found lots of different bugs
Partner configuration: lots of stuff that was previously only available on e.g. Translate Wiki, Varnish, ...
Yuri set up interface for partners
Erik: this is the "Zero:" namespace on Meta - pretty cool

Kul:
working on implementations
in some countries this is 90% of the people we can reach
Pilot goals: market test, analytics, qualitative feedback, server load and scale, UX and flow, continuity plans
keep it simple first, collect as much data as possible
Dan explains SMS / USSD:
customer dials access code, gets question form (USSD), types query (USSD), selects from dynamically formed results list (people are used to interfaces like that), gets article on separate screens or receives as concatenated SMS
Sue: is this a real search interface, or do i have to know the exact article title?
Yuri: Praekelt (in South Africa) uses the same search API as the desktop site
Dan: real search
Dan: demonstrates South African pilot service
search "light". ... receive SMS with first part, "reply for more"
Sue: could be painful user experience ;)
Dan: this demo is via roaming, faster when based in ZA itself
Kul: people are used to this
Frank: it's still a different user experience. this might mean that the lede of WP articles becomes more important
Erik: this is all free, including the "more" message? yes
Erik: reason for USSD/SMS distinction?
Dan: ... 2nd reason: SMS are stored in your phone, can go back to them later. USSDs are not
Kul: USSD login session times out
Kul: Legal is still working on this
Dan: Practically, Praekelt acts as middleman between our servers and the carrier
Erik: so they are basically the service provider? (responsible for service quality)
Kul: basically yes
aiming at three month public pilot starting by July 31
Dan: another reason for staggering pilots: learn from first ones

Dan on Zero UX flow:
considering simplifying "data charges may apply" warning banner for external links:
Sue: this would be more like an interstitial? (i.e. interrupting flow until user makes decision)
Dan: on JavaScript phones, yes

WP0 work for other WMF projects?
Commons is whitelisted, but not other wikis
Amit, Dan: enabling all other language versions of WP could be complicated - some partners don't support wildcards in URLs

Kul:

We now know a lot about what we don't know ;)
Awareness of WP?
South Africa: good among students of certain age, but needs more research
surprisingly low among others we asked on the street, e.g. working class/middle class
higher in some poorer villages in Kenya - no libraries there (books get stolen), sometimes internet cafes are an option
Sue: my impression in e.g. India was awareness of WP high, but not as something they "own" (something American).
Kul: others absolutely have no idea what WP is
but people find idea appealing ("make knowledge come out of my phone too")
Similar with search - e.g Google Uganda has some awareness problems
Amit: formal research study:
behaviors norm and attitudes that may inhibit (or enourage) people in Africa to use WP

• 3 filmed focus groups in each of Botswana and Uganda
• 500 face to face interviews in each country ("man on the street")
• Mobile surveys: 4500 total in 9 countries (Vietnam, India, Indonesia, Philippines, Kenya, Nigeria, Uganda, South Africa, ...). Participants rewarded with airtime.
  

Sue: can we already give a "headline" for the results?
Amit: not yet until data comes in, but can summarize themes:

• credibility - a lot of deference to teachers, people still remember if they were told not to use WP 5y ago in school
• skepticism about free offers, not too apparent
  

Kul: heard from Google that this really differs by country in Africa
Frank: is this distrust to US things in general?
Kul: we can hypothesize about this, but don't really know if that's the case.
(Amit:)

• familiarity with encyclopedias, doesn't seem huge issue
• local relevance. E.g. existence of WikiProject Botswana content surprised people there, excitement about local language versions of WP
  

Sue: this sounds like a theme we could push with partners for marketing. People's default assumption might be that it's all (e.g.) American content, if they look at the WP front page. Should emphasize "it's for you" (eventually, also "by you")
Dan: at least the landing pages link to local language edition
(Amit:)

• Comparison with other media, like newspapers

--break--

(Amit:)
need to devote resources for awareness building

• Many current customers of partners still don't know mobile WP is free for them, even though they are frequent WP readers on desktop.
• some social media grassroots effect, low hanging opportunity
• explaining WP one to one is really effective, but hard to scale. empower chapters to promote program? case study?
  

Sue: problem: chapters underrepresented in Global South. perhaps use CentralNotice to reach out to Wikipedians in that country? create toolkit for outreach?
Kul: you can't teach passion
Erik: how do you collaborate with Communications team?
Kul: blog posts (https://blog.wikimedia.org/c/technology/mobile/wikipedia-zero/ ), press releases
consultants (Minassian) help with US press
we need help from experts in local markets
Sue: don't use if not useful
Amit: public evangelism (editorials etc)
Kul: could also have downsides when pushing too hard
Amit: should aim for two "kits" -- both to convince new partners and to educate public about programs that are already in place
Marketing with other mobile partners:

• Mozilla (Firefox OS), they advocated to get WP preloaded with some partners
• Opera Mini. but browser based environments declining in general
  

CentralNotice banners, e.g. "Wikipedia Zero is now available in Sri Lanka" (simple, no operator logos)
try next week, work with Philippe and J.Alexander
Frank: coordinate messaging with Communications
Sue: great idea, need not be provider-specific
Erik: could (should) easily test effectiveness
Kul: if we have case studies with good stats, this is really useful for partners
3 key issues:

• know what we dont know -> research
• sustainability
• we might be swimming against general market trends. other Zero programs scaling down in general, operators focusing more on monetization
  

Amit: Whatsapp (taking away SMS revenue) shocked many of them
Mobile PVs going down in general?
e.g. our Mobile PVs flat in some African countries last few months
overall environment might become more app-based

(Kul&Dan:)
New pipeline development

• support smaller partners
• USSD/SMS pilot
  

Dan: automated testing to catch issues due e.g. to code changes more quickly, improve communication
more stable portal for partners
Kul: we were a bit resource-constrained about this
Erik: coordination with partners on admininstrative/technical work and on e.g. marketing campaigns is separate
Dan: we are on our way with new Meta-based system

recap what was accomplished:

• reach is 410 million subscribers now, 228 million launched
• free access via text trials
• quadrupled pageviews
  

...

• making progress while fighting market forces

Erik: which positions still open for hiring?
Kul: nothing at the moment but Director position. This doesn't address our immediate needs
Frank: was taken out of plan, but some ongoing discussion about others
we are definitely underresourced on the awareness building/social media area
Kul: based on talking with consultants, my impression is knowledge of local markets useful
Sue: mainstream media might not be the way to go
however, Minassian might be able to help with case studies
Erik: goal is to increase beyond organic growth
seems to me need resources both on design and communications side
Sue: Communications felt capable and resourced to help with WP0, but felt hampered by a lack of strategy for WP0 some months ago - this might be different now
Now team knows more, could go to Communications with more specific needs
Erik: so no extra resources in that area?
Sue: not at this point
Kul: Analytics issues continued
Sue: What about Ops - team reduced dependency on them, right?
Dan: yes, working pretty well with them at this point
Kul: Legal problems caused externally (internally OK, could just streamline timelines a bit more)
Erik: Victor's work completely independent? (film, https://blog.wikimedia.org/2013/06/19/movement-for-free-access-to-wikipedia-south-africa/ etc.)
Kul: I do coordinate with him on some things but he makes many of those decisions on his own. I do not drive or control his agenda.
Erik: so communications support is main need at this point?
Kul: yes, and partner coordination
Sue: How do you feel now compared to last review? Last time numbers were dramatically lower than expected, but we and the Board felt this was OK because when we set the targets our understanding of how WP0 would or could work was extremely limited, and so "what success could reasonably be expected to look like" remained an open question. Now the numbers are up, due in part to new partnerships but also because we know we don't see pay-off of existing partnerships until the last step of the chain (deal signed > product launched > marketing builds awareness) and many of the existing partnerships have just begun to hit the paying-off point. So the numbers are up which is good. At the same time, I'm worried about the "swimming against the tide" thing. If selling WP0 is extremely difficult because it's less and less the kind of project carriers want to participate in, then we will want to remain open to rethinking. The ultimate goal is to grow readership in Global South (not merely numbers for this particular program), and whatever path leads most directly to that is the one we should be on. That said, we feel things are paying off, seeing successes on the ground. Right?
Kul: Yes
Erik: Need to watch percentage of WP among all mobile traffic. E.g. could be that data charges become less of an issue in the future.
How do we factor the data available now in dashboards into our monthly metrics review, goalsetting etc.? That's a conversation we need to have in a few weeks, with data stabilizing.
Kul: eventually WP0 could go away, which may or may not be a bad thing. Markets and user behavior changes all the time. Like USSD - it could be part of the tech/mobile timeline, although it could take longer than people think.
Erik: Tomasz, any other issues from the engineering side?
Tomasz: All have been brought up already, we are working pretty smoothly now.
Frank: Extra position that was dicussed a while ago?
Kul: off the table. still, missed lots of opportunities due to limited time
Dan: Yuri and Adam were productive within a month, other engineers might have taken much longer
first quarter was just firefighting, now fortunately can do more strategic work
Ken: as a newcomer, I found this quarterly review was really useful