Trisector would not be nearly as complete and polished without the help of a handful of really awesome testers. I was lucky enough to have a few good friends that loved video games and had the time and patience to help test my game during development.

I had a few devices to test on (iPhone4,1; iPhone5,1; iPad1,1; iPad2,5) but didn’t have a Retina iPad to use for testing. I enlisted Swain, my first tester (and eventually his trusty iPad3,3), at the start of December 2013 during the tail end of engine development. It was a good thing he had an iPad3,3 because Trisector continually found new ways to make that device “hitch” during play (also iPad 3,1 and 3,2 were similarly affected).

The term hitch came to mean the number of times that the device ran longer than the allotted 16.67ms per frame (60fps), which caused a noticeable jump in the game field scrolling. It was easy enough to determine when the hitch happened during play. If the amount of movement of the foreground terrain layer was more than a threshold number of pixels, then that was a hitch. Early versions of the game counted the number of hitches encountered during play and then displayed the hitch count on the Game Over screen.

Engine development was wrapping up around mid December 2012. Swain had been testing on his iPhone4S which was running Trisector smoothly. I finally got the engine to support both iPad/iPhone on the same code base and finished some placeholder iPad Retina assets. There were still a lot of free CPU/GPU cycles on the iPad Mini (iPad2,5), and I figured that the beefy iPad3,3 wouldn’t have any problems. However, the iPad3,3 seriously choked on Trisector, and lit up the hitch-o-meter (as it came to be called).

The hitch issue was eventually resolved with some heavy refactoring of the rendering engine and the collision detection. Without Swain’s testing and feedback through this refactoring, I don’t think the hitch issue would have been resolved (or even found) without the purchase of an iPad3,1. The hitch issue resurfaced a few months later when the background layer’s Blur Shader was implemented, which also caused the iPad3,3 to light up the hitch-o-meter again.

With the hitching issues mostly in past, focus turned to game play and design. Enemies and their bullets were implemented in early January 2013 with a few placeholder enemies, and fully implemented into the types seen in the v1.0 final by mid February 2013. Power-ups, Art, SFx, Stats, Leaderboards, Achievements, Scrolling Playfield, etc. were added from January to early March. Early March also introduced the first implementation of the 8 Levels of the game (which later became 9), and the implementation of a few additional testers to the game.

There were a few times along the way were a tester had an old build and reported an issue that was already fixed in a newer build. Thus, code was written to automatically increment the build numbers, which could then be displayed on the title screen. In order to auto increment the build numbers in Xcode, the build number was set to a starting numerical value such as 100. A script was then added to the build phase to increment this number when the application was built under Release as an Archive:

Here’s the incrementing build number script for easy cutting and pasting. Note, that the “Run script only when installing” checkbox means to only run this script when performing an Archive. That way the build number isn’t incremented every time the code is built and run under a Release configuration.

#!/bin/bash
# NOTE: checkbox for only when installing means run only on Archived
if [ "${CONFIGURATION}" = "Release" ]; then
  buildNumber=$(/usr/libexec/PlistBuddy -c "Print CFBundleVersion" "$INFOPLIST_FILE")
  buildNumber=$(($buildNumber + 1))
  /usr/libexec/PlistBuddy -c "Set :CFBundleVersion $buildNumber" "$INFOPLIST_FILE"
fi

The version and build number were added to the main title scene as follows, which ensured that the testers always knew which version and build of the app they were using.

// build number format: v1.2.3 (150)
NSString *buildString = [NSString stringWithFormat:@"v%@ (%@)",
                        [[[NSBundle mainBundle] infoDictionary] objectForKey:@"CFBundleShortVersionString"],
                        [[[NSBundle mainBundle] infoDictionary] objectForKey:@"CFBundleVersion"]];

And the display of the version and build number in a near final version:

Before additional testers were enlisted, the manual reporting of hitches, deaths, etc had to be automated in order to receive accurate test data. Since the game was already collecting stats for each level such as number of hitches, Power-ups collected, Enemies Shot, Walls Shot, etc., it wasn’t much more work to have the game automatically send the level data to a server for collection.

I created a data collection and reporting class called the UsageTracker. It’s operation was pretty simple, in that at the end of each attempt on a level (either after a victory, failure, restart, etc), the StatsTracker class would instantiate a UsageTracker class which would asynchronously send a formatted GET request to a stats collection server. Since this is mostly used for debug and visualizations, it’s not important that it always gets delivered so the UsageTracker is more fire and forget.

The core of the UsageTracker was simply creating the URL request, asynchronously sending the request, and checking the request response. Once it was completed or timed out (e.g. USAGETRACKER_TIMEOUT_ASYNC is 500ms or so), then the UsageTracker class would be cleaned up.

// set up the request
NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:url
                                                       cachePolicy:NSURLRequestReloadIgnoringCacheData
                                                   timeoutInterval:USAGETRACKER_TIMEOUT_ASYNC];

// tell the server what to expect
[request addValue:@"application/x-www-form-urlencoded" forHTTPHeaderField: @"Content-Type"];

// set as get request
[request setHTTPMethod:@"GET"];

// make the connection to the server
[NSURLConnection sendAsynchronousRequest:request
                                   queue:[NSOperationQueue mainQueue]
                       completionHandler:^(NSURLResponse *response, NSData *responseData, NSError *error){

        // get the response code
        NSHTTPURLResponse *responseCode = (NSHTTPURLResponse*) response;

        // do additional logic as needed

}]; // end block

The server side would simple accept the request, parse/verify the input, and stick the input into various database tables. For example, the usage data contains device information such as a unique Trisector device identifier (created via CFUUIDCreate), device model, and build number, and also includes level specific information for each level attempt such as level number, level difficulty, number of hitches, level percent, level time, final player position, number of each type of power-up collected, number of shots fired, etc.

Using the level attempt data, it was easy to determine which devices were hitching and on which levels. One discovery from this information was that the tutorial levels were making certain devices (I’m looking at you iPad3,1 - iPad3,3) hitch. Since the tutorials don’t have much in the way of a foreground terrain layer, the hitching wasn’t easily perceived, but the usage data told the real story. It turns out that the help messages that are displayed on the screen during the tutorial levels were really taxing. All of the help messages were a CCLabelTTF, and each time a new message was displayed, a new CCLabelTTF was created (which was really expensive in this situation). I swapped the CCLabelTTF out with a CCLabelBMFont that was merely updated each time the message changed, which resolved the hitching issue.

As an added bonus, swapping to a CCLabelBMFont allowed the icons of the power-ups to be added into the font files and displayed along with the help text, which helped better associate the icon with the description.

With the latest round of hitching issues resolved, I was able to utilize the level attempt data to try to figure out where in the levels the testers were having trouble. Trisector was designed to be a hard game, but I still wanted the first few levels to somewhat ease the player in and help prepare them for levels 5-9. By plotting player positions in the level attempt against a png of the foreground layer of a level, it became pretty apparent that the original tutorial level was crushingly hard. A lot of the new testers didn’t even make it out of the tutorial level. Around the same time, a few new testers were enlisted, and some of them (e.g. Ken and Chris) provided some really good feedback on what was confusing in the game.

With the level attempt data and the tester feedback, the original tutorial level was scrapped, and the tutorial was broken into 3 levels. The first tutorial level is pretty heavy on the game play information, so it was designed so it was impossible/near impossible to die on. The second tutorial was an introduction to the enemy ships and some of the tactics used to fight them, and is hopefully difficult to die on. The third tutorial was more representative of the actual game, and was overly populated with power-ups and easy enemies. The third tutorial is pretty short because I expected some deaths on it and didn’t want to make the players slog through 2 minutes of a monolithic tutorial only to die at the very end.

In the end, I got lucky with my group of testers as the majority of them provided some really great feedback. Not only was their feedback clear, but by some stroke of luck each tester had reports focused on a different area of the game.

Swain really did an awesome job testing from start to finish, and had great feedback on almost every aspect of the game (e.g. performance, controls, design, sfx, music, tutorials, etc.). I doubt Trisector would have been half the game without his valuable input!

My brother Jeremy was the only tester on an iPod4,1, and had great feedback too. It was his feedback coupled with my own experiences on an iPhone4,1 that really cemented Trisector as an iPad only release at the start. The game runs great on an iPhone4S or better, but is a little choppy on anything lesser. Since I didn’t have a good way to exclude the problem devices, I’m keeping Trisector off those devices for the initial release. The eventual v1.0.1 Universal release finally had Trisector running fine on all devices.

Ken had a small amount of time testing but provided a huge amount of very detailed feedback. His feedback focused in on some problem areas in the UI, and really helped push me to improve those areas that I had simply deemed ‘good enough’ at some point (e.g. speed meter, gear icon, and some power-up icons). Also, his confusion on some core mechanics (e.g. shield deactivation / reactivation) really helped push me to break the old tutorial into 3 parts.

Chris picked up steam at the end and provided some really great feedback on the controls. He found a few edge cases for tilt control and firing input that really helped to improve the controls. Also, his feedback helped make the tutorials clearer as well.

My bro-from-another-mo-in-law Michael helped with general play testing and was also a reason for making the improved tutorial.

A big thanks to everyone who helped test Trisector along the way. I had this game kicking around in my head and various notebooks for years, and it was great to have all of the help and feedback in refining Trisector. Thanks guys!

Hope you found this article on Trisector’s Testing interesting,

Jesse from Smash/Riot