Getting calculated NA column values ​​in julia DataFrames that respond to dropna

Question

I am trying to use NAas a result to indicate that the calculated value for a given row of data "row" does not make sense (or, perhaps, can not be calculated). How to get a calculated column NAthat is still responding to dropna?

Example:

using DataFrames

df = DataFrame(A = 1:4, B = [1, 0, 2, 3], C = [5, 4, 3, 3])

# A value of 0 in column B should yield a foo of NA
function foo(d)
  if d[:B] == 0
    return NA
  end
  return d[:B] ./ d[:C] # vectorized to work with `by`
end

# What I'm looking for is something equivalent to this list
# comprehension, but that returns a DataFrame or DataArray
# since normal Arrays don't respond to `dropna`

comprehension = [foo(frame) for frame in eachrow(df)]  

Answer 1

You can do it...

using DataFramesMeta
result = @with(df, map(foo, :B, :C)) 

#=> DataArray{Any,1}: [0.2, NA, 0.667, 1.0]

... if foo you can rewrite to refer to individual values, and not to the whole DataFrame:

function foo(b, c)
  if b == 0
    return NA
  end
  return b / c
end

Similarly, if you want the new DataFrame to contain a new column, use @transform:

tdf = @transform(df, foo = map(foo, :B, :C))
#=>4x4 DataFrame
#  | Row | A | B | C | foo      |
#  |-----|---|---|---|----------|
#  | 1   | 1 | 1 | 5 | 0.2      |
#  | 2   | 2 | 0 | 4 | NA       |
#  | 3   | 3 | 2 | 3 | 0.666667 |
#  | 4   | 4 | 3 | 3 | 1.0      |

Answer 2

Base.convert DataArrays.dropna, dropna Vector s:

using DataFrames

function Base.convert{T}(::Type{DataArray}, v::Vector{T})
  da = DataArray(T[],Bool[])
  for val in v
    push!(da, val)
  end
  return da
end

function DataArrays.dropna(v::Vector)
  return dropna(convert(DataArray,v))
end

:

df = DataFrame(A = 1:4, B = [1, 0, 2, 3], C = [5, 4, 3, 3])

# A value of 0 in column B should yield a foo of NA
function foo(d)
  if d[:B] == 0
    return NA
  end
  return d[:B] / d[:C]
end

comprehension = [foo(frame) for frame in eachrow(df)]  

dropna(comprehension) #=> Array{Any,1}: [0.2, 0.667, 1.]

dropna convert DataFrame DataArray, NA :

conv = convert(DataArray, comprehension)
insert!(df, size(df, 2) + 1, conv, :foo)
#=> 4x4 DataFrame
#  | Row | A | B | C | foo      |
#  |-----|---|---|---|----------|
#  | 1   | 1 | 1 | 5 | 0.2      |
#  | 2   | 2 | 0 | 4 | NA       |
#  | 3   | 3 | 2 | 3 | 0.666667 |
#  | 4   | 4 | 3 | 3 | 1.0      |

typeof(df[:foo]) #=> DataArray{Any,1} (constructor with 1 method)
dropna(df[:foo]) #=> Array{Any,1}: [0.2, 0.667, 1.]

Answer 3

, - . , , :

using DataFrames
df = DataFrame(A = 1:4, B = [1, 0, 2, 3], C = [5, 4, 3, 3])

# A value of 0 in column B should yield a foo of NA
function foo(d)
    if d[:B] == 0
    return NA
  end
  return d[:B] / d[:C] # vectorized to work with `by`
end
comp = DataArray(Float64,4)
map!(r->foo(r), eachrow(df))

`map!` has no method matching map!(::Function, ::DFRowIterator{DataFrame})

, by, , - :

using DataFrames
df = DataFrame(A = 1:4, B = [1, 0, 2, 3], C = [5, 4, 3, 3])

# A value of 0 in column B returns an empty array
function foo(d)
    if d[1,:B] == 0
        return []
  end
    return d[1,:B] / d[1,:C] #Plan on only getting a single row in the by
end

by(df, [:A,:B,:C]) do d
    foo(d)
end

3x4 DataFrame
| Row | A | B | C | x1       |
|-----|---|---|---|----------|
| 1   | 1 | 1 | 5 | 0.2      |
| 2   | 3 | 2 | 3 | 0.666667 |
| 3   | 4 | 3 | 3 | 1.0      |