Deliver 1.2 Million Records from iPhone App

Question

I have a data set of 1.2 million key-value pairs. Keys are strings (a sequence of numbers up to 22 characters long), and values ​​are strings.

What is the best way to send this value quickly and quickly?

I suspect that plist is not suitable for a data set of this size.

I have a data set that is stored in two ways: CSV and mySQL database table with two columns. I ll go ahead using any method that best uses the data in the application.

Answer 1

a set of text files may work well. You can:

• divide them into several files (for example, by character range).
• order of pairs (e.g. by character number)
• and quick / easy to read step by step / parts as needed.
• balance resources between reading files and using memory.
• choosing the right encoding for strings can also help (I would start with utf8 if it is mostly in ascii).

If the size of the distribution is also your problem, you can compress / unzip these files.

or you can just use this approach and use your own serialized class to represent subsets of the collection if it sounds like too much parsing and reading.

If you use objc types for storage and / or parsing, then it would be nice to keep these files small. if you use c or C ++, this will help profile the application.

your data set will require up to 30 MB using a single-byte encoding with 8 bits per character. one large file (again, ordered), which you will also consider. see [NSData initWithContentsOfMappedFile:path]; .

Answer 2

Basic data and SQLite are two good options for working with very large datasets in iOS. It's easy to create a Core Data model for the data you're talking about. You can then copy this model into a small command-line program that you will write to move the data to the master data warehouse. Then you can move the resulting data file to iOS application resources.

A third option, especially useful if the data can change frequently, is to create a web service to provide data from the service. I do not think that this is what you are asking for, but something needs to be considered if the data set is very large and / or subject to frequent changes.

Answer 3

My personal experience is with the plist file, which has only thousands of entries, and I can say that it is not so fast. Therefore, my options for this amount of data you have:

• Database.
• Or, if you have sorting criteria for these keys and prefer to plist files break it into many files and save a reference dictionary with a key to run each file. E.g. all keys starting with "abc" go to a.plist, etc.

(I don’t know if this applies to your application, but you can consider transferring data to the server and searching through a web service, especially if your data will grow.)

Answer 4

The sqlite file is probably the best choice. You can create it on the desktop using the sqlite3 command line or any sqlite gui. Make sure you specify the key column.

Import the csv file as described here: Import CSV files into sqlite

Then just add the database to the project / target. However, if you want to change the database at run time, you will have to copy it to your documents or cache directories.

For wrapping objective-c around sqlite i like fmdb