Assuming what monthis in the class Date(or similarly for POSIXt, IDateTimeor other classes with a method diff), you can use the function diffto do this.
setkeyv(dt, c("seller", "buyer", "month"))
dt[, start := c(TRUE, diff(month) > 31), by = list(seller, buyer)]
dt[, end := c(diff(month) > 31, TRUE), by = list(seller, buyer)]
EDIT: @ : , , . , .
dt[, ":=" (start = c(TRUE, diff(month) > 31),
end = c(diff(month) > 31, TRUE)),
by = list(seller, buyer)]
EDIT2: , : , start = c(TRUE, ...). , (31 ), diff(month) > 31. , c(TRUE, diff(month) > 31).
, , .