Pre-aggregated data structure in clojure

Question

In OLAP cubes, you can do a very quick search for large amounts of aggregated data. The main reason for this is that one pre-aggregates the data in operations that are easy to combine up (basically +, -, mean, std, max, min and a few more).

How to get this "anti-lazy" behavior in clojure?

I think of something like

(def world-population {:africa 4e8 ;;this is an aggregation! :africa/liberia 3.4e6 :africa/ethiopia 7.4e7 ...}) 

How to update such a data structure and make sure that the object’s parents are also updated? Do I need to steer my own ref implementation?

Answer 1

You can write a recursive convolution function as a function of a higher order, for example:

 (defn rollup ([data heirarchy func] (loop [top (second (first heirarchy))] (if (nil? (heirarchy top)) (rollup data heirarchy func top) (recur (heirarchy top))))) ([data heirarchy func root] (let [children (reduce (fn [l [kv]] (if (= v root) (cons kl) l)) '() heirarchy) data (reduce (fn [dc] (if (dc) d (rollup d heirarchy func c))) data children) child-values (map data children)] (assoc data root (apply func child-values))))) 

which can then be used with any particular drilling operation or hierarchy you like:

 (def populations { :africa/liberia 3.4e6 :africa/ethiopia 7.4e7}) (def geography {:africa/liberia :africa :africa/ethiopia :africa :africa :world}) (rollup populations geography +) => {:africa 7.74E7, :world 7.74E7, :africa/ethiopia 7.4E7, :africa/liberia 3400000.0} 

Obviously, it becomes more complex if you have very large data sets or several hierarchies, etc., but this should be enough for many simple cases.

Answer 2

By storing data in the atom, you can add hours - in fact, callbacks when updating the atom.

Something like that:

 (def world-population (atom {:africa 4e8 :africa/liberia 3.4e6 ...})) (add-watch word-population :population-change-key (fn [key ref old new] (prn "population change"))) 

Instead, you can build some event propagation logic.