BlinkUp App Testing: Strategies and Best Practices

Electric Imp’s BlinkUp™ technology and all BlinkUp Software Development Kit (SDK) features are tested by the Electric Imp Quality Assurance Department. However, the mobile BlinkUp applications you create using Electric Imp’s SDKs also need testing for all your custom development components and features, in addition to testing overall integration with BlinkUp. This guide helps explain the scope of testing for your mobile BlinkUp app, touches on QA best practices, and recommends some test scenarios.

Contents

1. Testing Scope
   • Custom Content
     • Customer Account Creation and Management
     • Screens, Pages and Transition Flows
     • Product Interactions
     • App Settings
     • Error Scenarios and Related Messaging
     • Device Compatibility
     • Localization
     • Usability
   • Electric Imp BlinkUp Integrations
     • Pre-BlinkUp
     • BlinkUp
     • Post-BlinkUp
     • Plan ID
     • BlinkUp Webhook
     • Clear Device Configuration
2. QA Best Practices
   • Mobile Devices and Operating System Compatibility
   • App Store Submission Requirements
   • Test Automation
   • Load Testing
   • Third-Party Integrations
   • Security
3. Test Scenarios
   • As a Customer...
     • ...I want to get my purchased product online so that I can use it
     • ...I want to use my connected product so that it satisfies the intended purpose
     • ...I want to reconfigure my already connected product in order to use new WiFi settings
     • ...I want to reconfigure the same product that was previously used by a different customer in order to own the product myself
   • As a company stakeholder...
     • ...I want to view data about our shipped products in order to better understand customer usage
   • Generic Test Case Examples

Testing Scope

Both your app content and integrations with Electric Imp components need testing throughout the development process. The following sections list testing requirements and categorize them into two main sections: your custom content and Electric Imp BlinkUp integrations.

Custom Content

Customer Account Creation and Management

• New account creation from within app and backend data correctly maintained for accounts.
• Login and logout, auto login and account switching.
• Change account name, email and/or password from within the app.

Screens, Pages and Transition Flows

• Content should correctly fit as desired with the proper resolution for different mobile device screen sizes and rotation orientations. This is challenging because of the large device variety in the marketplace, as visualized in this 2015 Android device fragmentation report. More information on supporting multiple Android screen sizes is available at the Android developer center. View layout compatibility testing should first focus on most common top percentages and your target market, followed by outliers for edge-case scenarios.
• Each page should appear correctly and have the ability to transition to all other pages in every possible flow allowed by the app (forward, backward and different link paths).

Product Interactions

• Controlling the connected product from within the app and all possible options, such as turning on or off.
• Viewing within the app data that has been sent from the connected product.

App Settings

• Changing app-level options, such as display name, volume, color, layout, time zone, language, and anything else customizable within the app.
• Changing product settings, such as temperature, timed events, display text, voice, favorites, and anything else customizable for product interactions.
• Persistence of customer settings, such as app closing/opening or app deletion and re-installation (if saved server side in relation to customer account).

Error Scenarios and Related Messaging

• Connection loss and inability to interact with your backend server.
• Invalid account credentials at login, or account credentials changed/invalidated while user already logged into the app.
• Failed BlinkUp screens and retry flow, for example from invalid WiFi credentials.
• App exit during expected process, returning to app after action failed.
• Backend server temporarily down/outage, resulting in unexpected client errors.
• OS-level notification interruptions.

Device Compatibility

• App installation on different hardware models and operating systems.
• App features working correctly on different hardware models and operating systems.
• App content and text displaying correctly on different hardware models and operating systems.

The Electric Imp Dev Center has more information regarding SDK minimum specifications.

Localization

• All strings are correctly displayed in relation to device language settings.

Usability

• In-app guide and training helps new users navigate the process of getting your product online, and how to use the product after connecting. For example, screenshots of the product’s phototransistor location, process of power cycling to enter BlinkUp recognition mode, and where to hold the mobile device during BlinkUp.
• Input/Button locations and natural transition flows between screens.
• Warn users not to look at the mobile screen during BlinkUp, especially those sensitive to flashing or strobing lights.

Electric Imp BlinkUp Integrations

Pre-BlinkUp

Before BlinkUp flashing begins, users need a way to configure their connectivity settings. It is these settings that will be sent by BlinkUp to the imp-enabled device.

• All desired user options should be available: for example, SSID, WiFi password, WPS pin, static IP and proxy settings.
• Sensitive data, eg. password, should not be displayed in the clear unless user chooses.
• If your app allows for saving BlinkUp user settings, they should persist for subsequent BlinkUps.

BlinkUp

BlinkUp functionality itself is tested and verified by Electric Imp, but your app could potentially interfere with the flashing process, thus causing BlinkUp failures. This is typically the result of app activity causing BlinkUp to drop frames.

• Warn users not to look at the mobile screen during BlinkUp, especially those sensitive to flashing or strobing lights.
• Suspend background threads during BlinkUp.
• Ensure the presenting view does not re-draw during BlinkUp.

Post-BlinkUp

After the BlinkUp flashing there is a polling process which repeatedly checks the Electric Imp impCloud™ for information about the imp-enabled device’s registration. Either the device connects to the impCloud and information is successfully retrieved, or the polling will fail after a timeout period.

• Keep the user informed while the app is waiting for the enrollment polling process to complete.
• Success messaging and transition to next steps (main menu routing, product usage tutorial etc).
• Failure messaging (for example, invalid WiFi credentials) and retry flow.

Plan ID

This identifier differentiates one end-user from another. An end-user’s plan ID is generated when they first configure a production product using BlinkUp. Your app design chooses whether the customer’s plan ID is retained throughout ownership of the product (based on their account in your backend system), or if it is re-generated every time the product needs to be re-configured, for example when the customer changes their local WiFi network details. More information on Plan IDs is available in the Dev Center.

• Every BlinkUp using correct same plan ID per account, and a new Plan ID when using a different account. Or using a different plan ID every BlinkUp regardless of account. Expected test results for consecutive BlinkUps in relation to account depend on your design choices.
• Your backend system correctly storing plan IDs in relation to each of your customer accounts and their configured products.

BlinkUp Webhook

This event-triggered webhook is called when a customer successfully configures a product using BlinkUp. More information on Webhooks is available in the Dev Center.

• Webhooks are correctly configured as desired, such as correct schema, domain, path, and setting for JSON encoded or URL encoded payload.
• Your backend system correctly receives and stores incoming BlinkUp webhook data from Electric Imp.
• Ideally this endpoint of yours would also be tested under load, in case multiple customers perform BlinkUp simultaneously.

Clear Device Configuration

This action clears network access credentials and enrollment data from an imp-enabled device. You may or may not include this feature in your application. An example use case might be someone re-selling their product and wanting to first clear their product data.

• Messaging for post flashing screen with info on success or failure (no server response, imp light pattern indicates result).
• Navigation routing options as desired, such as back to main menu.

QA Best Practices

Creating a test plan and testing strategy, in addition to continuously testing throughout the entire development process, are two examples of relatively easy best practices that pay big dividends when ensuring a desired level of product quality within project timelines. The following examples of QA best practices are specifically relevant for mobile testing.

Mobile Devices and Operating System Compatibility

• The mobile device marketplace is extremely fragmented, resulting in a wide variety of differences with regards to screen sizes, performance levels, and OS versions. Each of these aspects can make a significant difference on the user experience and functionality within your application.
• Both Apple’s Xcode Simulator and Google’s Android Studio Emulator provide a way of testing applications across different mobile devices.
• In order to complete end-to-end testing, you will also need physical devices for successful SDK BlinkUp transitions. Our recommendation is to test with a few different physical devices in addition to the simulator/emulator. Please note the SDK supported devices for a list of the many devices known to work well with BlinkUp.
• Screen orientation should be tested on every page for potential layout issues. Also verify screen orientation changes across different screen sizes.
• Test on different OS versions (easier with simulator/emulator) in order to verify no compatibility issues. It is also a good idea to sanity test your already released application when new OS versions are available.
• App should be able to install both clean and as an update (if applicable) for different OS versions and device hardware models.

App Store Submission Requirements

• Technical content and design criteria should be considered prior to submitting an app for review approval. Apple Review Guidelines and Google Play Launch Checklist are available to help navigate the requirements and provide guidance for distribution approval.
• Consider design differences between iOS and Android, such as back button support on Android.

Test Automation

• Adding test automation will help provide continuous and efficient test coverage while preventing regression bugs during iterative development. Manual testing and automated testing should be used together as part of an overall solution for measuring product quality.
• Both Apple’s Xcode UI Automation instrument, and Google’s Android Studio UI Automator and Espresso testing framework provide ways to automate user interface tests through test scripts running outside of your app code.
• Many companies provide mobile test automation services and there are also open-source options such as Appium and Selenium WebDriver automation libraries.

Load Testing

• Load testing is an important process for determining a system’s behavior under normal and anticipated peak usage conditions.
• Example mobile client scenarios inducing server load are as follows:
  • Account creation.
  • Login.
  • Updating user settings.
  • Interacting with connected product (if device’s agent sends data to your backend system), synchronized timing events for device reports.
  • BlinkUps. The BlinkUp action adds load to your system if Electric Imp Webhooks are configured, since we will notify your backend for each successful BlinkUp.

Third-Party Integrations

• Functionality of integration with external services, such as analytics, crash reporting, custom push notifications and advertising.
• Potential performance related issues and impacts.
• Suspend background threads and ensure presenting view does not re-draw during BlinkUp, since these could cause a few frames to drop while flashing.

Security

• Both client and server-side data storage should be encrypted. For example, store passwords within an encrypted data section in the iOS keychain or within encrypted storage in internal app data directory for Android.
• Assume that your client app will not be the only thing accessing your backend APIs. This means the server that your mobile app is accessing, along with all your APIs, should be verified and have security measures in place to prevent unauthorized users from accessing data.
• Look for unintended data leakage, such as through integrated analytics providers or advertising. This should be relatively easy to spot by viewing network traffic in a proxy.

Test Scenarios

Each application will have its own unique test cases as related to specific individual aspects within the application. These test scenarios provided here are meant to provide an example high-level concept of what to test. In order to provide a bit more context, the format of these test scenarios borrow concepts from user stories and behavior-driven development acceptance criteria. All these test scenarios will have multiple test cases and particular test steps in relation to your application. In addition, a few example generic test-case considerations are also provided at the end.

As a customer, I want to get my purchased product online so that I can use it

• Given a new blessed product available for connecting.
• When a customer installs the mobile app for the first time.
• And uses the app to connect their product.
• Then all app screens should transition properly (create account, login, user options, WiFi setup, post BlinkUp).
• And the product should become connected online.

As a customer, I want to use my connected product so that it satisfies the intended purpose

• Given a connected product and mobile app already authenticated.
• When a customer uses the app features to interact with the product.
• Then all product features should work as intended (controlling the product from app, viewing product data on app, etc).

As a customer, I want to reconfigure my already connected product in order to use new WiFi settings

• Given new WiFi credentials that make a previously connected product now unable to connect.
• When a customer uses the app to reconnect their product.
• Then the app transition workflow should allow successfully reconnecting the same product with new WiFi credentials.

As a customer, I want to reconfigure the same product that was previously used by a different customer in order to own the product myself

• Given a connected product by Customer A.
• When a different Customer B logs into the app to connect the same product.
• Then Customer B should be able to connect the product and use it as their own.
• And Customer A should no longer be able to see/use the same connected product.

As a company stakeholder, I want to view data about our shipped products in order to better understand customer usage

• Given multiple distributed blessed devices and a subset of customer BlinkedUp/enrolled devices.
• When a company stakeholder with proper access rights views data about the shipped products on own company's existing internal integrated tracking system.
• Then data from Electric Imp’s Blessing and BlinkUp webhooks should appear as desired.
• And any configured integrations with Electric Imp agents should show relevant communicated data, such as factory test results and ongoing connected product information.
• And any configured data integrations from the mobile application itself should appear as expected.

Generic Test Case Examples

• Attempt to perform BlinkUp with invalid WiFi credentials, failure flow should allow fixing and successful retry.
• Lose mobile device connectivity while within the app.
• Cancel all screens, for example using UI elements or the back button on Android.
• Close/Open the app from different screens.
• OS notifications interrupting app actions (and any other OS priority actions).
• Attempt to login with invalid user and/or password (wrong, too long, too short, invalid character, empty field, etc).
• Login to mobile app, change password somewhere else (such as correlating customer facing web app), and then attempt to continue using the mobile app.
• Same login and app usage on multiple apps at the same time.
• Server side content updates to the client (if applicable).