Reading GPS RINEX data with Pandas

Question

I am reading the observational data file [RINEX-3.02] (p. 60) in order to perform some filtering based on the satellite identifier by time, and will eventually restore it later. This would give me more control over the choice of satellites that I allow to contribute to the decision of the position over time with the subsequent processing of RTK.

In particular, for this part, I just use:

• [Python-3.3]
• [pandas]
• [NumPy]

Here is an example with the first three observations with time stamps.
Note. I do not need to parse the data from the header.

3.02 OBSERVATION DATA M: Mixed RINEX VERSION / TYPE CONVBIN 2.4.2 20130731 223656 UTC PGM / RUN BY / DATE log: /home/ruffin/Documents/Data/in/FlagStaff_center/FlagStaCOMMENT format: u-blox COMMENT MARKER NAME MARKER NUMBER MARKER TYPE OBSERVER / AGENCY REC # / TYPE / VERS ANT # / TYPE 808673.9171 -4086658.5368 4115497.9775 APPROX POSITION XYZ 0.0000 0.0000 0.0000 ANTENNA: DELTA H/E/N G 4 C1C L1C D1C S1C SYS / # / OBS TYPES R 4 C1C L1C D1C S1C SYS / # / OBS TYPES S 4 C1C L1C D1C S1C SYS / # / OBS TYPES 2013 7 28 0 27 28.8000000 GPS TIME OF FIRST OBS 2013 7 28 0 43 43.4010000 GPS TIME OF LAST OBS G SYS / PHASE SHIFT R SYS / PHASE SHIFT S SYS / PHASE SHIFT 0 GLONASS SLOT / FRQ # C1C 0.000 C1P 0.000 C2C 0.000 C2P 0.000 GLONASS COD/PHS/BIS END OF HEADER > 2013 7 28 0 27 28.8000000 0 10 G10 20230413.601 76808.847 -1340.996 44.000 G 4 20838211.591 171263.904 -2966.336 41.000 G12 21468211.719 105537.443 -1832.417 43.000 S38 38213212.070 69599.2942 -1212.899 45.000 G 5 22123924.655 -106102.481 1822.942 46.000 G25 23134484.916 -38928.221 656.698 40.000 G17 23229864.981 232399.788 -4048.368 41.000 G13 23968536.158 6424.1143 -123.907 28.000 G23 24779333.279 103307.5703 -1805.165 29.000 S35 39723655.125 69125.5242 -1209.970 44.000 > 2013 7 28 0 27 29.0000000 0 10 G10 20230464.937 77077.031 -1341.254 44.000 G 2 20684692.905 35114.399 -598.536 44.000 G12 21468280.880 105903.885 -1832.592 43.000 S38 38213258.255 69841.8772 -1212.593 45.000 G 5 22123855.354 -106467.087 1823.084 46.000 G25 23134460.075 -39059.618 657.331 40.000 G17 23230018.654 233209.408 -4048.572 41.000 G13 23968535.044 6449.0633 -123.060 28.000 G23 24779402.809 103668.5933 -1804.973 29.000 S35 39723700.845 69367.3942 -1208.954 44.000 > 2013 7 28 0 27 29.2000000 0 9 G10 20230515.955 77345.295 -1341.436 44.000 G12 21468350.548 106270.372 -1832.637 43.000 S38 38213304.199 70084.4922 -1212.840 45.000 G 5 22123786.091 -106831.642 1822.784 46.000 G25 23134435.278 -39190.987 657.344 40.000 G17 23230172.406 234019.092 -4048.079 41.000 G13 23968534.775 6473.9923 -125.373 28.000 G23 24779471.004 104029.6643 -1805.983 29.000 S35 39723747.025 69609.2902 -1209.259 44.000 

If I need to make my own parser,
Another tricky thing is that satellite identifiers come and go over time,
(as shown with satellites "G 2" and "G 4")
(plus they also have spaces in identifiers)
Since I read them in a DataFrame,
Do I need to make new column labels (or row labels for MultiIndex?) When I find them.

I initially thought this could be considered a MultiIndex problem,
but I'm not sure pandas read_csv can do anything
Go to reading DataFrame objects with MultiIndex

Any suggestions?

Relevant sources, if interested:

• [python-3.3]: http://www.python.org/download/releases/3.3.0/
• [numpy]: http://www.numpy.org/
• [the pandas]: http://pandas.pydata.org/
• [RINEX-3.02]: http://igscb.jpl.nasa.gov/igscb/data/format/rinex302.pdf
• [ephem]: https://pypi.python.org/pypi/ephem/
• [RTKLIB]: http://www.rtklib.com/
• [NOAA CORS]: http://geodesy.noaa.gov/CORS/

Answer 1

That's what I finished doing

 df = readObs(indir, filename) df.set_index(['%_GPST', 'satID']) 

Please note that I just installed the new MultiIndex at the end after creating it.

 def readObs(dir, file): df = pd.DataFrame() #Grab header header = '' with open(dir + file) as handler: for i, line in enumerate(handler): header += line if 'END OF HEADER' in line: break #Grab Data with open(dir + file) as handler: for i, line in enumerate(handler): #Check for a Timestamp lable if '> ' in line: #Grab Timestamp links = line.split() index = datetime.strptime(' '.join(links[1:7]), '%Y %m %d %H %M %S.%f0') #Identify number of satellites satNum = int(links[8]) #For every sat for j in range(satNum): #just save the data as a string for now satData = handler.readline() #Fix the names satdId = satData.replace("G ", "G0").split()[0] #Make a dummy dataframe dff = pd.DataFrame([[index,satdId,satData]], columns=['%_GPST','satID','satData']) #Tack it on the end df = df.append(dff) return df, header 

Using a dummy data frame just doesn't look the most elegant though.

Answer 2

I suggest writing your own parser by reading the file line by line.

The space in "G 5" is another tip for writing a custom parser.
In this case, you cannot separate the arguments with just space,
you must immediately read all 3 characters and delete the first char and convert the remaining two ("5") to the number of satellites.