Learn the Best Practices in Mobile Game Testing презентация

Learn the Best Practices in Mobile Game Testing 2 December 2014 Ville-Veikko Helppi Technical Product Manager [email protected]

Agenda Different Types of Testing and How Game Testing Differs from Regular App Testing? Frameworks, Tools and Technologies – What is Important in Mobile Game Testing Today? Top Priorities for Mobile Game Testing Infrastructure for Successful Game Testing Hands-on Example (Image Recognition) Testdroid Update Q&A

Different Types of Testing? Black-box testing vs. White-box testing Methods used in Mobile Game Testing Functional Compatibility Performance Localization Regression Load Smoke Acceptance

Defining Some Ways to Test

The Old-Fashion Thinking Deterministic systems Identifiable objects Tool support(s) User-driven actions Simple inputs/outputs Virtualization

Test Automation as a Solution

Challenges that Test Automation Tackles Proliferation of Devices, Brands, Operators, Platforms Coupling the Testing and Development process (Continuous Integration/Deployment/Delivery) Different Types of Testing (as automated) Stress, Load, Performance, Functionality etc. Test script & test asset usability Cons of Manual testing Tedious effort, Expensive human resources, Error-prone (as done by humans) & Not reusable (done once)

Significant Benefits of Using Automation

Different 'Verticals' Require Different Practices

Important Aspects in Game Testing User Interface and Functionality UI layouts/elements, Menus, Resolution, Orientation, Overall visuality Graphics Performance Usability and User Experience Responsiveness, Real Occuring Events Multi-player/User Social Integrations Security and Liabilities

User Interface and Functional Compared to regular app testing, mobile game testing may seem to be more challenging e.g. OpenGL ES content not action-identifiable Graphics and UI – Identical look&feel across all devices and variants Usability controls in the context of game Navigation/game-play, Progress, Delays between "game-stages" Animations, Fonts, Other graphics assets Screen Resolution and Orientation

Performance Testing Performance, Load, Stress, Spike, Soak

Performance Makes +/- UX Studies: Majority of users leave game due sluggish, general slowness or other bad performance behavior It's very important to understand how game behaves on different devices, under different circumstances

Building the User Experience

Multi-Player/User Features End-to-end between tests, AI or more simple implementations Optimize the data communication between server and clients Test simultaneously on several (different) devices Check what happens if game gets interrupted (platform) Location-based features – how to properly mock up those? Hermetic environment – what happens?

Social Integrations Important part of the after-engagement (how gamers share their results) Many of OEM branded devices provide certain SoMe apps pre-installed (version?) Connectivity, login and data transferring are tested in context of game Hermetic environment – what happens?

Security and Liabilities

Localization

Device Diversity

Real Devices vs. Emulators Emulators/simulators won’t help you to test… User Experience and Usability Hardware Software Infrastructure

Manual vs. Automation

High-Level Infrastructure

Architecture / Infrastructure

Isolated, Hermetic Testing

Compatibility & Porting Testing

Smoke/Reference Testing

Back-End Performance Testing

Continuous Integration

Game Testing Process

Setup Continuous Integration (e.g. Jenkins) Appium framework (for Android and iOS) No need for devices – we'll use devices on Testdroid Cloud service Test scripts / test cases for game

Continuous Integration

Appium Intro

Appium Basics Uses Selenium Webdriver (W3C standard) as a scripting framework Supports native Android, native iOS and mobile web: Android via uiautomator (API level >=16) and Selendroid (API level <16) iOS via UI Automation Mobile web as Selenium driver for Android and iOS You can write your Appium scripts on almost any programming language (Haskell/Go/Clojure/Java/Ruby)

Appium @ Localhost

Appium @ Localhost

Appium @ Localhost

Appium @ Localhost

Appium @ Localhost

Projects and Test Runs

Appium @ Testdroid

Appium @ Testdroid

Appium @ Testdroid

Appium @ Testdroid

Appium @ Testdroid

Running tests on multiple devices

Appium @ Localhost

Example: Clash of Clans Test Script contains the following 'steps': Identify the used platform (Android / iOS) Search for 'Goldmine' graphical element (using find_image to search for screen) Script runs test to go to shop Script runs test to buy a cannon Script runs test to place the cannon (all these three are defined in .png files) Script runs test to start a battle! Game is brought down and test ends.

THANK YOU! Video recording & slides of this webinar will be soon available at Vimeo and Slideshare! Please visit at www.testdroid.com for more information.