r - Append a column of NA values: lit() and withColumn() giving error -
i trying append column of null values sparkr dataframe following code:
w <- rbind(3, 0, 2, 3, na, 1) z <- rbind("a", "b", "c", "d", "e", "f") x <- rbind(3, 3, 3, 3, 3, 3) d <- cbind.data.frame(w, z, x) b <- as.dataframe(sqlcontext, d) b1 <- sample(b, withreplacement = false, fraction = 0.5) b2 <- except(b, b1) col_sub <- c("z", "x") b2 <- select(b2, col_sub) b2 <- withcolumn(b2, "w", lit(na))
but, last expression returns error: error in fun(x[[i]], ...) : unsupported data type: null
. have used lit
operation produce column of null values before, i'm not sure why won't work time.
also, has been discussed on se before, see this question. i'm clueless why expression yields error. reference, i'm using sparkr 1.6.1.
no matter if works or not adding column way not practice. since practical reason add column contains undefined values enforcing specific schema unions or external writes should use columns of specific type.
for example:
withcolumn(b2, "w", cast(lit(null), "double"))
Comments
Post a Comment