When to apply (pd.to_numeric) and when to astip (np.float64) in python?

Question

I have a pandas DataFrame object named xiv that has an int64 column of int64 Dimensions.

 In[]: xiv['Volume'].head(5) Out[]: 0 252000 1 484000 2 62000 3 168000 4 232000 Name: Volume, dtype: int64 

I read other posts (like this and this ) that offer the following solutions. But when I use any approach, it does not change the dtype underlying data:

 In[]: xiv['Volume'] = pd.to_numeric(xiv['Volume']) In[]: xiv['Volume'].dtypes Out[]: dtype('int64') 

Or...

 In[]: xiv['Volume'] = pd.to_numeric(xiv['Volume']) Out[]: ###omitted for brevity### In[]: xiv['Volume'].dtypes Out[]: dtype('int64') In[]: xiv['Volume'] = xiv['Volume'].apply(pd.to_numeric) In[]: xiv['Volume'].dtypes Out[]: dtype('int64') 

I also tried to make a separate pandas Series and use the methods listed above in this series and reassign to the object x['Volume'] , which is the object pandas.core.series.Series .

However, I found a solution to this problem using the numpy package float64 type - , but I don't know why it is different .

 In[]: xiv['Volume'] = xiv['Volume'].astype(np.float64) In[]: xiv['Volume'].dtypes Out[]: dtype('float64') 

Can someone explain how to do with the pandas library what the numpy library seems to do easily with its class float64 ; i.e. convert the column in xiv DataFrame to float64 in place.

Answer 1

If you already have numeric dtypes ( int8|16|32|64 , float64 , boolean ), you can convert them to another "numeric" dtype using the Pandas .astype () method.

Demo version:

 In [90]: df = pd.DataFrame(np.random.randint(10**5,10**7,(5,3)),columns=list('abc'), dtype=np.int64) In [91]: df Out[91]: abc 0 9059440 9590567 2076918 1 5861102 4566089 1947323 2 6636568 162770 2487991 3 6794572 5236903 5628779 4 470121 4044395 4546794 In [92]: df.dtypes Out[92]: a int64 b int64 c int64 dtype: object In [93]: df['a'] = df['a'].astype(float) In [94]: df.dtypes Out[94]: a float64 b int64 c int64 dtype: object 

This will not work for object (string) dtypes that cannot be converted to numbers:

 In [95]: df.loc[1, 'b'] = 'XXXXXX' In [96]: df Out[96]: abc 0 9059440.0 9590567 2076918 1 5861102.0 XXXXXX 1947323 2 6636568.0 162770 2487991 3 6794572.0 5236903 5628779 4 470121.0 4044395 4546794 In [97]: df.dtypes Out[97]: a float64 b object c int64 dtype: object In [98]: df['b'].astype(float) ... skipped ... ValueError: could not convert string to float: 'XXXXXX' 

So here we want to use the pd.to_numeric () method :

 In [99]: df['b'] = pd.to_numeric(df['b'], errors='coerce') In [100]: df Out[100]: abc 0 9059440.0 9590567.0 2076918 1 5861102.0 NaN 1947323 2 6636568.0 162770.0 2487991 3 6794572.0 5236903.0 5628779 4 470121.0 4044395.0 4546794 In [101]: df.dtypes Out[101]: a float64 b float64 c int64 dtype: object 

Answer 2

I have no technical explanation for this, but I noticed that pd.to_numeric () causes the following error when converting the string 'nan':

 In [10]: df = pd.DataFrame({'value': 'nan'}, index=[0]) In [11]: pd.to_numeric(df.value) Traceback (most recent call last): File "<ipython-input-11-98729d13e45c>", line 1, in <module> pd.to_numeric(df.value) File "C:\Users\joshua.lee\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\tools\numeric.py", line 133, in to_numeric coerce_numeric=coerce_numeric) File "pandas/_libs/src\inference.pyx", line 1185, in pandas._libs.lib.maybe_convert_numeric ValueError: Unable to parse string "nan" at position 0 

whereas astype (float) does not have:

 df.value.astype(float) Out[12]: 0 NaN Name: value, dtype: float64