KIF: How do I automatically run / stress test an iOS app to find the cause of a rare UI error?

Question

Note. . I added kif to the header, just to search for an indexing navel, considering that most of the answer was for discussion.

I am looking for something like selenium for iOS, mainly for the test automation system / unit test, which can run a specific user interface script many times until it works, which will help me narrow down the cause of the user interface error, this happens very rarely and randomly.

(and by the way, I NSLogged every line of the code for interacting with the data source / table and spent hours analyzing the potential reason .. but I did not find anything convincing .. again this error very rarely happens).

I looked at some of the unit testing modules in iOS , but there seem to be a lot of them. I'm not sure which one to choose. Also my link to selenium is based on a hypothesis, since I worked with QA people who have used Selenium in large web projects in the past (and I assume there should be something similar for iOS).

Now that I am a one-person team working on an iOS project, I will need to put on a QA hat and calculate this error.

I came across a classic error that occurs when there is a discrepancy between the actual number of rows inserted in the UITableView and the number of rows returned by the data source delegate. This error message is:

*** Assertion failure in -[UITableView _endCellAnimationsWithContext:] Exception in insertRows: Invalid update: invalid number of rows in section 0. The number of rows contained in an existing section after the update (2) must be equal to the number of rows contained in that section before the update (2), plus or minus the number of rows inserted or deleted from that section (1 inserted, 0 deleted) and plus or minus the number of rows moved into or out of that section (0 moved in, 0 moved out). 

I click on a UITableViewCell which brings me to another UITableView . Sometimes it works

and sometimes (very rarely) it is not (with the error above):

Answer 1

update:. I added example code about KIF 2.0 at the bottom after the separator. for those who are more interested in KIF than the specific problem I am facing:

After some research and experimentation. I narrowed down my options to two test automation libraries: Frank and KIF . In the end, I decided to use KIF when borrowing the Gherkin cucumber syntax to describe my unit tests.

The reason I chose KIF (not Frank ) was because KIF based on 100% obj-c, instead of using ruby, as well as in the case of Frank . Thus, the setup is simpler, and it is more applicable to my narrow exam requirement. In doing so, I admit that Frank would be more useful if my application were more complex (i.e. using intput from multiple servers, etc.). You can see the last quarter of this great presentation to learn more about the pros and cons of KIF, Frank, and other test automation systems, including Apple Automation UI .

After using KIF, I found the error causing the error above, and I was able to reproduce it using KIF 100% of the time! The reason this happened so rarely was because it only happened when I quickly shot through the screens. And since KIF automates the steps, they do it at an incredibly fast speed .. which detected an error :).

So the following example will be an example of the code I used for testing. This is just to give you a quick idea of ​​what KIF (and Gherkin) can do for you:

in one file, I specify the scripts that I want to run:

 - (void)initializeScenarios; { [self addScenario:[KIFTestScenario scenarioToCompleteSignInAndLoadInbox]]; [self addScenario:[KIFTestScenario scenarioToFillAttachmentsWithData]]; [self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucket]]; [self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucketSubView]]; } 

each script displays steps (to learn more about outline syntax - and behavioral development based on test driver development, I highly recommend reading this excellent book on cucumber ):

 /* @given the application is at a fresh state @and the user already has an imap email account with a valid username/pwd @then the user can successfully log in @and the inbox view will be loaded @and the inbox will get loaded with the latest batch of emails in the user inbox */ + (id)scenarioToCompleteSignInAndLoadInbox { KIFTestScenario *scenario = [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully log in."]; [scenario addStepsFromArray:[KIFTestStep stepsCompleteSignInAndLoadInbox]]; return scenario; } /* @given that the user is already signed in @and the user has already downloaded their folders @then the user can click on the folders view @and the user can click on the 'attachments' remote folder @and the latest batch from the 'attachments' remote folder will download */ + (id)scenarioToFillAttachmentsWithData { KIFTestScenario* scenario = [KIFTestScenario scenarioWithDescription:@"Test that we can view the attachments folder and fill it with data."]; [scenario addStepsFromArray:[KIFTestStep stepsToFillAttachmentsWithData]]; return scenario; } /* @given that the user is already signed in @and the user has already downloaded their folders @and the user has already downloaded attachments @then the user can click on inbox menu button @and the user can click on folder list menu button @and the user can click on the file bucket icon (on the account list view) @and the data for the file bucket is fetched from the dbase @and the file bucket view displayes the attachments */ + (id)scenarioToViewAndLoadFileBucket { KIFTestScenario *scenario = [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully view and load file bucket parent view"]; [scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketPage]]; return scenario; } /* @given that the user is already signed in @and the user has already downloaded their folders @and the user has already downloaded attachments @and the user has already opened file bucket view @then the user can click on a random row in the file bucket view table @and the subview will retrieve data from the dbase pertaining to that row @and the subview will display the data in the uitableview */ + (id)scenarioToViewAndLoadFileBucketSubView { KIFTestScenario *scenario = [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully view and load filet bucket sub view"]; [scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketSubPage]]; return scenario; } 

and steps are defined using KIF UI automation methods (this is just one example):

 // this step assumes there is an attachment folder that contains emails with attachments + (NSArray *)stepsToFillAttachmentsWithData { NSMutableArray* steps = [@[] mutableCopy]; [steps addObject: [KIFTestStep stepToTapViewWithAccessibilityLabel:@"InboxMenuButton"]]; NSIndexPath* indexPath = [NSIndexPath indexPathForRow:remoteAttachmentFolderNumber inSection:0]; KIFTestStep* tapAttachmentRowStep = [KIFTestStep stepToTapRowInTableViewWithAccessibilityLabel: @"attachments" atIndexPath:indexPath]; [steps addObject:[KIFTestStep stepToWaitForNotificationName: (NSString *)kBeganSyncingOlderEmails object:nil whileExecutingStep:tapAttachmentRowStep]]; [steps addObject:tapAttachmentRowStep]; [steps addObject: [KIFTestStep stepToWaitForViewWithAccessibilityLabel:@"attachments"]]; KIFTestStep *fillingInboxStep = [KIFTestStep stepToWaitForNotificationName: (NSString *)kOldMailBatchDelivered object:nil]; [fillingInboxStep setTimeout:kSpecialTimeoutForLongTests]; [steps addObject:fillingInboxStep]; return steps; } 

---

KIF 2.0 code example: KIF 2.0 uses Xcode 5 an all-new test navigator .. which is a big improvement than what KIF 1.0 did .. now your tests feel much more organic and natural than the past .. (that is, it goes in real time, and does not create scripts that run in the future, etc.). You can even check each of them using the play button, etc. you should try.

Here are some examples (again, using gherkin syntax):

 #import <KIF/KIF.h> #import "KIFUITestActor+EXAdditions.h" #import "KIFUITestActor+UserRegistration.h" @interface LoginTests : KIFTestCase @end @implementation LoginTests - (void)testReset { [tester flushDbase]; [tester reset]; } /* @given that the app is in a fresh clean state @and that no one has ever registered with the server @then the user can register their themselves with the server @and immediately start with the rider map @and their location on the map shows */ - (void)testRegistration { [tester flushDbase]; [tester reset]; [tester singleUserRegistration]; [tester showUserCurrentLocationOnMap]; } /* @given that the user has already registered with the server @and the user is not currently logged in @then the user can login using their user name and password @and immediately start with the rider map @and their location on the map shows */ - (void)testSuccessfulLogin { [tester reset]; [tester login]; [tester showUserCurrentLocationOnMap]; } /* @given that the user has already registered @and that the user is already logged in before app launch @then the user starts on the map view with the location visible @and the button prompts them to set pick up location */ - (void)testStartOfApplication { [tester showUserCurrentLocationOnMap]; [tester showsPickUpButton]; } @end 

here is the implementation of some test cases in category files:

 - (void)reset { [self runBlock:^KIFTestStepResult(NSError **error) { BOOL successfulReset = YES; // Do the actual reset for your app. Set successfulReset = NO if it fails. AppDelegate* appDelegate = [[UIApplication sharedApplication] delegate]; [appDelegate resetApp]; KIFTestCondition(successfulReset, error, @"Failed to reset some part of the application."); return KIFTestStepResultSuccess; }]; } - (void)flushDbase { [self runBlock:^KIFTestStepResult(NSError **error){ NSURL *url = [NSURL URLWithString:@"http://randomdomain.com/flush_db"]; NSURLRequest *request = [NSURLRequest requestWithURL:url]; NSError *connectionError = nil; BOOL databaseFlushSucceeded = YES; NSURLResponse *response; NSData *resultData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&connectionError]; if (!resultData) { databaseFlushSucceeded = NO; KIFTestCondition(databaseFlushSucceeded, error, @"failed to connect to server!"); } if (connectionError) { databaseFlushSucceeded = NO; KIFTestCondition(databaseFlushSucceeded, error, [NSString stringWithFormat:@"connection failed. Error: %@", [connectionError localizedDescription]]); } return KIFTestStepResultSuccess; }]; } - (void)navigateToLoginPage { [self tapViewWithAccessibilityLabel:@"login email"]; } - (void)returnToLoggedOutHomeScreen { [self tapViewWithAccessibilityLabel:@"Logout"]; [self tapViewWithAccessibilityLabel:@"Logout"]; // Dismiss alert. }