How to serialize / deserialise long [] value from get / set to random indexes using chronicle map?

Question

I am new to the chronicle map. I am trying to simulate a heap map using a chronicle map, where the key is a primitive short and the value is a primitive long array. The maximum size of a long array value is known for this map. However, I will have several cards of this type, each of which may have a different maximum size for the long value of the array. My question is about serializing / deserializing the key and value.

From reading the documentation, I understand that for the key, I can use the ShortValue value type and reuse the implementation instance of this interface. Regarding the value that I found, the page talks about DataAccess and SizedReader , which gives an example for byte [], but I'm not sure how to adapt it along []. Another additional requirement: I need to get and set values ​​for arbitrary indices in a long array, without having to pay all the serialization / deserialization of the whole value every time.

So my question is: how can I model the type of value when building a map and what serialization / deserialization code is needed for a long [] array if the maximum size is known for each map, and I need to be able to read and write random indices without serializing / deserializing all the useful load values ​​every time? Ideally, long [] will be encoded / decoded directly to / from the heap without undergoing intermediate heap conversion in byte [], and the chronicle card code will not be allocated at runtime. Thanks.

Answer 1

Firstly, I recommend using the unstructured LongList abstraction instead of long[] , this will simplify the work with size variability, provide alternative versions of flyweight, etc.

If you want to read / write only individual elements in large lists, you should use the extended context API :

 /** This method is entirely garbage-free, deserialization-free, and thread-safe. */ void putOneValue(ChronicleMap<ShortValue, LongList> map, ShortValue key, int index, long element) { if (index < 0) throw throw new IndexOutOfBoundsException(...); try (ExternalMapQueryContext<ShortValue, LongList, ?> c = map.getContext(key)) { c.writeLock().lock(); // (1) MapEntry<ShortValue, LongList> entry = c.entry(); if (entry != null) { Data<LongList> value = entry.value(); BytesStore valueBytes = (BytesStore) value.bytes(); // (2) long valueBytesOffset = value.offset(); long valueBytesSize = value.size(); int valueListSize = (int) (valueBytesSize / Long.BYTES); // (3) if (index >= valueListSize) throw new IndexOutOfBoundsException(...); valueBytes.writeLong(valueBytesOffset + ((long) index) * Long.BYTES, element); ((ChecksumEntry) entry).updateChecksum(); // (4) } else { // there is no entry for the given key throw ... } } } 

Notes:

• You must acquire writeLock() from the very beginning, because otherwise readLock () will be automatically received when you call the context.entry() method, and you cannot update the read lock for writing later. Please read the HashQueryContext javadoc carefully.
• Data.bytes() officially returns RandomDataInput , but you can be sure (it is listed in Data.bytes() javadoc) that it is actually an instance of BytesStore (this combination of RandomDataInput and RandomDataOutput ).
• Assuming the correct SizedReader and SizedWriter (or DataAccess ). Note that the byte / element size method used is the same as the example in the SizedReader and SizedWriter section of the doc , PointListSizeMarshaller . You can create your LongListMarshaller in this class class.
• This listing is listed, see ChecksumEntry javadoc and the section on checksums in the document . If you have a purely chronological map (not saved) or the checksum is turned off, this call can be omitted.

The implementation of reading a single item is similar.

Answer 2

Answering additional questions:

I implemented SizedReader + Writer. Do I need DataAccess or SizedWriter fast enough for primitive arrays? I looked at ByteArrayDataAccess, but it is not clear how to port it for long arrays, given that the internal HeapBytesStore is so specific for the [] / ByteBuffers byte?

Using DataAccess instead of SizedWriter allows SizedWriter to make one copy of data of a lower value on Map.put(key, value) . However, if in your use case putOneValue() (as in the example above) is the dominant type of request, this will not make much difference. If Map.put(key, value) (and replace() , etc., T. E. Any "full write value" operations) are important, it is still possible to implement DataAccess for LongList . It will look like this:

 class LongListDataAccess implements DataAccess<LongList>, Data<LongList>, StatefulCopyable<LongListDataAccess> { transient ByteStore cachedBytes; transient boolean cachedBytesInitialized; transient LongList list; @Override public Data<LongList> getData(LongList list) { this.list = list; this.cachedBytesInitialized = false; return this; } @Override public long size() { return ((long) list.size()) * Long.BYTES; } @Override public void writeTo(RandomDataOutput target, long targetOffset) { for (int i = 0; i < list.size(); i++) { target.writeLong(targetOffset + ((long) i) * Long.BYTES), list.get(i)); } } ... } 

For efficiency, the size() and writeTo() methods are key. But it is also important to correctly implement all other methods (which I did not write here). Carefully read the DataAccess , Data and StatefulCopyable javadocs, as well as Understanding StatefulCopyable , DataAccess and SizedReader and Custom Serialization Checklist in the tutorial with great attention too.

---

Is a read / write lock a mediator when reading and writing multiple processes on one computer or in only one process?

This is process safe, note that the interface is called InterProcessReadWriteUpdateLock .

---

When storing objects with a variable size not known in advance, how will the values ​​cause fragmentation from the heap and in the saved file?

Saving the value for the key once and not changing the size of the value (and not deleting the keys) after that will not lead to fragmentation violation. Resizing a value or deleting keys can cause external fragmentation. ChronicleMapBuilder.actualChunkSize() configuration allows you to trade between external and internal fragmentation. The larger the piece, the less fragmentation, and the more internal fragmentation. If your values ​​are significantly larger than the page size (4K), you can set an absurdly large block size and still have internal fragmentation related to the page size, since the Chronicle Map can use the lazy page layout feature on Linux.