Serializing a Java Object Without a StackOverflowError

Question

I have a rather large Java object, which is a graph with vertices and edges in memory. Each vertex has an ArrayList other vertices to which it is connected (and has a HashMap data structure, as well as for other purposes). A graph can have several thousand vertices and many other edges.

When trying to serialize a graph using the built-in Java serialization ( implements Serializable , etc.), I always run a StackOverflowError . Setting other graph attributes on transient does not help, nor does it increase the stack size (i.e. -Xss1g or -Xss512m ).

I would not have thought that I needed to create my own writeObject method, since ArrayList and HashMap already have their own implementations that cause serialization.

My question is: is there a way to serialize a large Java object already in memory without receiving a StackOverflowError ?

Change Here is the stack trace:

 Exception in thread "main" java.lang.StackOverflowError at java.lang.reflect.Method.invoke(Method.java:575) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at java.util.ArrayList.writeObject(ArrayList.java:570) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at java.util.ArrayList.writeObject(ArrayList.java:570) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at java.util.ArrayList.writeObject(ArrayList.java:570) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at java.util.ArrayList.writeObject(ArrayList.java:570) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at java.util.ArrayList.writeObject(ArrayList.java:570) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at java.util.ArrayList.writeObject(ArrayList.java:570) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at java.util.ArrayList.writeObject(ArrayList.java:570) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at java.util.ArrayList.writeObject(ArrayList.java:570) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at java.util.ArrayList.writeObject(ArrayList.java:570) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) // Many more lines after this 

Here is my Graph class:

 public class Graph implements Serializable { /** * */ private static final long serialVersionUID = -2632163054149021990L; private ArrayList<Vertex> vertices; private HashMap<Integer, Set<Vertex>> map; public Graph(int rowMax, int colMax) { map = new HashMap<Integer, Set<Vertex>>(); this.vertices = new ArrayList<Vertex>(); } public void connectVertices(Vertex u, Vertex v) { u.addNeighbor(v); v.addNeighbor(u); } // other unrelated methods after this 

And here is my Vertex class:

 public class Vertex implements Serializable { /** * */ private static final long serialVersionUID = 8520500010710631610L; public int row; public int col; private ArrayList<Vertex> neighbors; // may change this to Set<Vertex> public Vertex(int i, int j) { this.row = i; this.col = j; this.neighbors = new ArrayList<Vertex>(); } public boolean addNeighbor(Vertex v) { this.neighbors.add(v); return true; } // unrelated methods after this 

Change 2 . In addition, for graphs of smaller sizes, but having "neighbors", this problem is not.

Answer 1

Serialization will StackOverflowError if the depth of your chart is too large to handle serialization by default. This is due to the default serialization, the recursive serialization of each node as it analyzes your graph.

Flat structures will work fine (for example, a parent node with 2000 children), but deep structures will not work (for example, a node with 2000 descendant levels).

eg. The following item will overflow:

 public class Node implements Serializable { private ArrayList<Node> nodes = new ArrayList<Node>(); public static void main(String[] args) throws Exception { Node node = new Node(); int depth = 3000; // Add nodes chained down to specified depth Node last = node; for (int i = 0; i < depth; i++) { Node temp = new Node(); last.nodes.add(temp); last = temp; } System.out.println("starting"); ByteArrayOutputStream bos = new ByteArrayOutputStream(); ObjectOutputStream out = new ObjectOutputStream(bos); // Below line will cause a stack overflow. out.writeObject(node); System.out.println("done"); } } 

You need to either reduce the depth of your graph to limit the number of recursive serialization calls, or write custom serialization to get around this. Custom serialization must be non-recursive in nature, and, unfortunately, at first glance it seems non-trivial to implement.

Answer 2

I had a similar problem. After a big hunt, I found a Cryo fork designed to handle deeply embedded objects. Via https://github.com/EsotericSoftware/kryo/issues/103 , clone and mvn clean install https://github.com/romix/kryo/tree/kryo-2.23-continuations . Currently com.esotericsoftware.kryo:kryo:2.23-SNAPSHOT .

(Please note: there are a number of questions on SO that can be answered. If I send copies of the answer (for example, fooobar.com/questions/507033 / ... ) or comment on the link to this answer or mark the questions as duplicate, indicating this , or what?)