How to use OrientDB ETL to create edges only

Question

I have two CSV files:

First contains ~ 500M records in the following format

identifier name
10000023432, Tom User
13943423235, Blah Man

The second containing the relationship ~ 1.5B each other in the following format

fromId, toId
10000023432,13943423235

I used the OrientDB ETL tool to create vertices from the first CSV file. Now I just need to create faces in order to establish a friendly connection between them.

I have tried several json ETL file configurations so far, the last one is:

{ "config": {"parallel": true}, "source": { "file": { "path": "path_to_file" } }, "extractor": { "csv": {} }, "transformers": [ { "vertex": {"class": "Person", "skipDuplicates": true} }, { "edge": { "class": "FriendsWith", "joinFieldName": "from", "lookup": "Person.id", "unresolvedLinkAction": "SKIP", "targetVertexFields":{ "id": "${input.to}" }, "direction": "out" } }, { "code": { "language": "Javascript", "code": "print('Current record: ' + record); record;"} } ], "loader": { "orientdb": { "dbURL": "remote:<DB connection string>", "dbType": "graph", "classes": [ {"name": "FriendsWith", "extends": "E"} ], "indexes": [ {"class":"Person", "fields":["id:long"], "type":"UNIQUE" } ] } } } 

But unfortunately, this also creates a vertex with the "from" and "to" attributes in addition to creating an edge.

When I try to delete a vertex transformer, the ETL process throws an error:

 Error in Pipeline execution: com.orientechnologies.orient.etl.transformer.OTransformException: edge: input type ' com.orientechnologies.orient.core.record.impl.ODocument$1$1@40d1 3 6a8' is not supported Exception in thread "OrientDB ETL pipeline-0" com.orientechnologies.orient.etl.OETLProcessHaltedException: Halt at com.orientechnologies.orient.etl.OETLPipeline.execute(OETLPipeline.java:149) at com.orientechnologies.orient.etl.OETLProcessor$2.run(OETLProcessor.java:341) at java.lang.Thread.run(Thread.java:745) Caused by: com.orientechnologies.orient.etl.transformer.OTransformException: edge: input type ' com.orientechnologies.orient.core.record.impl.ODocument$1$1@40d1 36a8' is not suppor ted at com.orientechnologies.orient.etl.transformer.OEdgeTransformer.executeTransform(OEdgeTransformer.java:107) at com.orientechnologies.orient.etl.transformer.OAbstractTransformer.transform(OAbstractTransformer.java:37) at com.orientechnologies.orient.etl.OETLPipeline.execute(OETLPipeline.java:115) ... 2 more 

What am I missing here?

Answer 1

You can import edges using these ETL transformers:

 "transformers": [ { "merge": { "joinFieldName": "fromId", "lookup": "Person.id" } }, { "vertex": {"class": "Person", "skipDuplicates": true} }, { "edge": { "class": "FriendsWith", "joinFieldName": "toId", "lookup": "Person.id", "direction": "out" } }, { "field": { "fieldNames": ["fromId", "toId"], "operation": "remove" } } ] 

The merge transformer joins the current csv line with the associated Person record (this is a bit strange, but for some reason it needs to be associated with the identifier with the original person).

The field transformer will remove the csv fields added by the merge section. You can try importing without a field transformer to see the difference.

Answer 2

With Java API you can read csv and then create edges

  String nomeYourDb = "nomeYourDb"; OServerAdmin serverAdmin; try { serverAdmin = new OServerAdmin("remote:localhost/"+nomeYourDb).connect("root", "root"); if (serverAdmin.existsDatabase()) { OrientGraph g = new OrientGraph("remote:localhost/"+nomeYourDb); String csvFile = "path_to_file"; BufferedReader br = null; String line = ""; String cvsSplitBy = " "; // your separator try { br = new BufferedReader(new FileReader(csvFile)); int index=0; while ((line = br.readLine()) != null) { if(index==0){ index=1; } else{ String[] ids = line.split(cvsSplitBy); String personFrom="(select from Person where id='"+ids[0]+"')"; String personTo="(select from Person where id='"+ids[1]+"')"; String query="create edge FriendsWith from "+personFrom+" to "+personTo; g.command(new OCommandSQL(query)).execute(); } } } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } finally { if (br != null) { br.close(); } } } } catch (IOException e) { e.printStackTrace(); }