to generate almost in real time, I would update the data in small packets, and also update the index in small packets every minute (it takes only a few seconds to update the index, depending on the size of the new data)
Remember to regularly optimize the index.
source share