All geek questions in one place

Create unique and slow merges

I play with a neo4j database of ~ 275,000 English words associated with the letters they contain. I am running Neo4j 2.0.1 Community Edition on Windows.

I am trying to use the following Cypher to insert new word nodes in a graph, update properties on these nodes and create new relationships with existing (letter) nodes if the new word is node:

BEGIN
MATCH (A:Letter {token:"A"}),
(B:Letter {token:"B"}),
(C:Letter {token:"C"}),
(D:Letter {token:"D"}),
(E:Letter {token:"E"}),
(F:Letter {token:"F"}),
(G:Letter {token:"G"}),
(H:Letter {token:"H"}),
(I:Letter {token:"I"}),
(J:Letter {token:"J"}),
(K:Letter {token:"K"}),
(L:Letter {token:"L"}),
(M:Letter {token:"M"}),
(N:Letter {token:"N"}),
(O:Letter {token:"O"}),
(P:Letter {token:"P"}),
(Q:Letter {token:"Q"}),
(R:Letter {token:"R"}),
(S:Letter {token:"S"}),
(T:Letter {token:"T"}),
(U:Letter {token:"U"}),
(V:Letter {token:"V"}),
(W:Letter {token:"W"}),
(X:Letter {token:"X"}),
(Y:Letter {token:"Y"}),
(Z:Letter {token:"Z"})
// Create Words and link to proper letters
MERGE (w1:Word {string:"WHOSE", length:5})
ON MATCH SET w1.s_enable1=TRUE
ON CREATE SET w1.s_enable1=TRUE
// create the letter->word relationships if necessary
CREATE UNIQUE (w1) <-[:IN_WORD {position:1}]- (W)
CREATE UNIQUE (w1) <-[:IN_WORD {position:2}]- (H)
CREATE UNIQUE (w1) <-[:IN_WORD {position:3}]- (O)
CREATE UNIQUE (w1) <-[:IN_WORD {position:4}]- (S)
CREATE UNIQUE (w1) <-[:IN_WORD {position:5}]- (E)
MERGE (w2:Word {string:"WHOSESOEVER", length:11})
ON MATCH SET w2.s_enable1=TRUE
ON CREATE SET w2.s_enable1=TRUE
CREATE UNIQUE (w2) <-[:IN_WORD {position:1}]- (W)
CREATE UNIQUE (w2) <-[:IN_WORD {position:2}]- (H)
CREATE UNIQUE (w2) <-[:IN_WORD {position:3}]- (O)
CREATE UNIQUE (w2) <-[:IN_WORD {position:4}]- (S)
CREATE UNIQUE (w2) <-[:IN_WORD {position:5}]- (E)
CREATE UNIQUE (w2) <-[:IN_WORD {position:6}]- (S)
CREATE UNIQUE (w2) <-[:IN_WORD {position:7}]- (O)
CREATE UNIQUE (w2) <-[:IN_WORD {position:8}]- (E)
CREATE UNIQUE (w2) <-[:IN_WORD {position:9}]- (V)
CREATE UNIQUE (w2) <-[:IN_WORD {position:10}]- (E)
CREATE UNIQUE (w2) <-[:IN_WORD {position:11}]- (R)
... N-2 more of these ...;
COMMIT
... M-1 more transactions ...

I use the neo4j shell to execute Cypher batch files like this to add new words. Most of the words MERGED already exist on the chart. Only a small part is new.

, : (a) (, 50 /50 N = 50) (b) ( CREATE UNIQUE), , " GC".

MERGE CREATE UNIQUE. ( ) Java Heap . (, - .)

, / , .

, , , Neoj4 : , , , (, , , ,...).

( neo4j.properties CREATE INDEX ON).

s_enable1 . "enable1" (173 122 ). sowpods (267,751 ). s_ "". , , , , ( ) . (, AA sowpods, enable1, AA node s_sowpods, s_enable1, TRUE.)

MERGE CREATE UNIQUE, , .

sowpods 2,5 () - [: IN_WORD] → (word) . enable1 500 K . ( enable1 , , 16-21 .)

- Windows 7. Java 7.51 x64. ( x32, 2 .) Java -XshowSettings 885.5 M max heap. , . ( ?)

+4
2

, /:

create constraint on (l:Letter) assert l.token is unique;
create constraint on (w:Word) assert l.string is unique;

, :

export word=WHOSE

MATCH (w:Word {string:{word}}) RETURN w;

, Neo4j .

- : WITH split ({word}, ​​"")

MERGE (w:Word {string:{word}, length:length({word})})
   ON CREATE SET w.s_enable1=TRUE
FOREACH (i in range(0,length({word})-1) | 
  MERGE (l:Letter {token:substring({word},i,1)})
  MERGE (l)-[:IN_WORD {position:i}]->(w)
)

:

MERGE (w:Word {string:"STACKOVERFLOW", length:length("STACKOVERFLOW")})
   ON CREATE SET w.s_enable1=TRUE
FOREACH (i in range(0,length("STACKOVERFLOW")-1) | 
  MERGE (l:Letter {token:substring("STACKOVERFLOW",i,1)})
  MERGE (l)-[:IN_WORD {position:i}]->(w)
)

: http://console.neo4j.org

+4

:

  • BatchInserter, . , , , , .
  • RAM- (tmpfs). ~ 7,5% , ~ 200 CREATE/s HDD ~ 1500 CREATE/s. SSD , , , HDD → SSD, , Neo4j. , 4 , 8 , tmpfs , HDD/Linux-/.
  • Cypher, - 10-50 MERGE Cypher. 50 MERGE s . 500-1000 MERGE Cypher Neo4j StackOverflowError.
  • number_of_CPUs - 2 Neo4j, 1 , - Neo4j. ~ 95% 8 i7-3770K.
  • . , (.. ConcurrentHashMap, ImmutableMap ), , . , Neo4j.
  • , . , , MATCH MERGE. in-memory , ImmutableMap, HashMap ConcurrentHashMap .

:

+2

Source: https://habr.com/ru/post/1535447/

More articles:


All Articles