This is a continuation of my previous question. How to create a lazy piecemeal sequence in clojure?
I want to download data from a database in parts. First, I load the first 500 lines, and then send a query to select from the next 500 lines, and so on, until I get all the data from the server.
I wrote the code:
(jdbc/atomic conn
(with-open [cursor (jdbc/fetch-lazy conn [sql_query])]
(let [lazyseq (jdbc/cursor->lazyseq cursor)
counter (atom 1)]
(swap! lazyseq_maps assoc :session_id {:get_next? (chan 1) :over_500 (chan 1) :data []})
(>!! (:get_next? (:session_id @lazyseq_maps)) true)
(go
(doseq [row lazyseq]
(swap! counter inc)
(when (<! (:get_next? (:session_id @lazyseq_maps)))
(swap! lazyseq_maps update-in [:session_id :data] conj row)
(if (not= 0 (mod @counter 500))
(>! (:get_next? (:session_id @lazyseq_maps)) true)
(>! (:over_500 (:session_id @lazyseq_maps)) true))))
;
(close! (:get_next? (:session_id @lazyseq_maps)))
(close! (:over_500 (:session_id @lazyseq_maps)))
(.close conn))
(when (<!! (:over_500 (:session_id @lazyseq_maps))) {:message "over 500 rows"
:id :session_id
:data (:data (:session_id @lazyseq_maps))}))))
I take strings using a dose loop. When the doseq has passed 500 lines, I leave the loop (when (<! (:get_next? (:session_id @lazyseq_maps)))
and wait for the signal from the outside to get the next 500 lines.
. , "Resultset is closed". Ie . , , go . ?