The reason you get this error message is because when using with=FALSE you pass data.table to handle j , as if it were a data file. Therefore, it expects a column vector, not an expression, which should be evaluated in j as new=.N .
From the documentation ?data.table about with :
By default, with=TRUE and j is evaluated inside the x frame; column names can be used as variables. When with=FALSE j is a column vector symbol or a numeric column vector at select, and the return value is always data.table.
When you use with=FALSE , you need to select the column names in j without . before () as follows: dt1[, (colShow), with=FALSE] . Other options: dt1[, c(colShow), with=FALSE] or dt1[, colShow, with=FALSE] . The same result can be obtained using dt1[, .(col3)]
To summarize: with = FALSE used to select columns using the data.frame method. So you have to do it as such.
Also, using by = colBy , you tell data.table to evaluate j , which contradicts with = FALSE .
From the documentation ?data.table about j :
One column name, one instance of column columns, list() of column name expressions, an expression or function call that evaluates a list (including data.frame and data.table, which are lists) or (when with=FALSE ) a vector of names or positions for Choose.
j is evaluated inside the data.table frame; that is, it sees the names of the columns as if they were variables. Use j=list(...) to return multiple columns and / or column expressions. One column or one expression returns this type, usually a vector. See examples.
See also paragraphs 1.d and 1.g of the data table introduction vignette .
ansvals is the name that is used in data.table. You can see where it appears in the code using ctrl + f (Windows) or cmd + f (macOS) here .