Master data search optimization

Question

I am working on a search function in one of my Core Data-based applications, and I'm trying to collect all the search optimization tips to get them as fast as possible. The search should be fast enough so that it can deliver almost instant results for a database of 20,000 objects.

What I have done so far (as optimization)

• Implemented the technique shown in session 137 of WWDC 2010, creating a keyword object and creating a relationship to many of my main object objects. The entity name attribute is indexed, and keywords are created during the initial import procedure, separating the corresponding lines in the main entities and normalizing them (devoid of cases and diacritics).
• Using >= and < binary comparators instead of BEGINSWITH etc. My predicate:

SUBQUERY(keywords, $keyword, ($keyword.name >= $LB) AND ($keyword.name < $UB)) .@count != 0

Where $LB is the string of lower bounds and $UB is the upper bound. I create a compound AND predicate using this format and an array of search terms.

Now I fetch one time (when the user types the first letter), using the batch size of about 20, and then narrowing down the search results using the NSArray -filteredArrayUsingPredicate method as they continue to print. I also prefetch the attitude of keywords because it is used for filtering. Obviously, the part that takes the most time is the initial sample. There is a noticeable delay of ~ 1-2s in the library of about 15,000 objects. Time profiling shows that actually sampling causes a delay:

http://cl.ly/3a1b2022452M2V323f2H

Another thing worth noting is that I have to get multiple objects for the results. All entities have a ranking attribute, but I can’t get it more than once, so I have to retrieve them separately, merge them into one array and sort manually using -sortedArrayUsingDescriptors .

Any advice on how to speed this up would be greatly appreciated.

EDIT: based on @ImHuntingWabbits suggestions:

After adding the KeywordFirstChar object KeywordFirstChar my data model (simplified) will look like this:

Now questions , how would I write a predicate for a Car object that selects based on KeywordFirstChar ? The only thing I can think of is this:

SUBQUERY(keywords, $keyword, $keyword.firstChar.char == %@) where %@ is the search character, but I don’t know how it would be much more efficient, given that it still needs to list keywords if I misinterpreted the sentences.

Answer 1

Your request is highly optimized, I think you have already taken many steps. As for the first character, you are mistaken.

You are still scanning 15k records for the first hit of the character and probably correspond to a large number of them.

You can optimize it by indexing your keyword index, creating two new objects:

• KeywordFirstChar
• KeywordFirstTwoChars

Both with a to-many relationship to the keywords they point to.

 if (searchPredicate.length == 1) { //search on KeywordFirstChar } else if (searchPredicate.length == 2) { //search on KeywordFirstTwoChars } else { //search on keyword } 

Thus, the table scan will go through a maximum of 26 and 676 rows, respectively, which should be pretty trivial. Just make sure the relationships are in the programmed request key chains in the fetch request, so you really get the data from disk.

Edit (object search):

You can follow the path of the relationship key, so it will be something like this:

  [fetchRequest setRelationshipKeyPathsForPrefetching:[NSArray arrayWithObject:@"keyword.sourceObject"]]; 

Where the keyword is the relation to the Keyword object, and sourceObject is the object that you want to end up with.

Edit (Predicate):

The predicate is essentially the same, just change the names to match the new entity (the name may not appear for the name, instead of firstChar or another property).