All geek questions in one place

How to import a large dataset into SAS?

I have a fairly large dataset in a text file with approximately 25 million rows and 200 columns (they are all numeric). I would like to run some summary statistics and data analysis (survival analysis).

  • What is the fastest way to import data into SAS ?

  • How much memory do I need for my PC to run such a large data set?

Since I'm new to SAS , any advice would be greatly appreciated.

+4
source share
1 answer

Caution: I never do this in SAS, but I think I understand how SAS works well enough to make some guesses.

  • I'm not sure that everything will be much faster than just reading your dataset when using PROC IMPORT. Specifying your information formats and formats in advance can help speed things up a bit, but PROC IMPORT only transfers them from the first 20 entries by default, so I would not want it to read your entire data set to figure out which data types to use. The fact that your columns are numeric will probably help you. The most important thing is to save the results in a permanent data set (i.e., specify a library for it) - if you need to import data only once, it does not really matter if it takes a lot of time.

  • One of the nice things about SAS is that, by default, data is stored on disk, not in memory, so the size of your RAM doesn't really limit the size of your data set. This may limit what you can do with this dataset, but I don't know enough about the internal SAS operations to be able to predict what you will encounter.

Hope this helps, but first of all, I would advise you to just start. You will quickly find out what SAS can and cannot do with your data.

+4
source

Source: https://habr.com/ru/post/1339471/

More articles:


All Articles