I want to split large xml into small pieces. I use VTDGen to split the xml file into small pieces, and it works fine for a file size <2 GB. VTD-xml uses IN-Memory to parse xml, where I don't want to load xml into memory. Therefore, I am trying to map memory using VTDGenHuge.
The code works fine with VTDGen, but when I VTDGenHuge does not work.
String prefix = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"+"\n";
String suffix = "\n</Employees>\n";
try {
VTDGenHuge vg = new VTDGenHuge();
if (vg.parseFile("C:\\Users\\abc\\Desktop\\latestxml\\Input_1.xml", true,VTDGenHuge.MEM_MAPPED)) {
int splitBy = ;
System.out.println("Started time"+ new Date());
VTDNavHuge vn = vg.getNav();
AutoPilotHuge ap = new AutoPilotHuge(vn);
ap.selectXPath("/Employees/Employee");
FastLongBuffer flb = new FastLongBuffer(4);
int i;
byte[] xml = vn.getXML().getBytes();
while ((i = ap.evalXPath()) != -1) {
flb.append(vn.getElementFragment());
}
int size = flb.size();
if (size != 0) {
File fo = null;
FileOutputStream fos = null;
for (int k = 0; k < size; k++) {
if (k % splitBy == 0) {
if (fo != null) {
fos.write(suffix.getBytes());
fos.close();
fo = null;
}
}
if (fo == null) {
fo = new File("C:\\Users\\abc\\Desktop\\Test\\xml\\"+"out" + k + ".xml");
fos = new FileOutputStream(fo);
fos.write(prefix.getBytes());
}
fos.write("\n".getBytes());
fos.write(xml, flb.lower32At(k), flb.upper32At(k));
}
if (fo != null) {
fos.write(suffix.getBytes());
fos.close();
fo = null;
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
I get the NUll value in "byte [] xml = vn.getXML (). GetBytes ();" when u do syso vn.getXML () you get the value of the object. but with "getBytes ()" return null.I don't know why. But if u "byteAt (x)" x = any long value, it returns a value.
My xml file:
<?xml version="1.0" encoding="UTF-8"?>
<Employees>
<Employee id="1">
<age>29</age>
<name>Pankaj</name>
<gender>Male</gender>
<role>Java Developer</role>
</Employee>
<Employee id="2">
<age>35</age>
<name>Lisa</name>
<gender>Female</gender>
<role>CEO</role>
</Employee>
<Employee id="3">
<age>40</age>
<name>Tom</name>
<gender>Male</gender>
<role>Manager</role>
</Employee>
<Employee id="1">
<age>29</age>
<name>Pankaj</name>
<gender>Male</gender>
<role>Java Developer</role>
</Employee>
<Employee id="2">
<age>35</age>
<name>Lisa</name>
<gender>Female</gender>
<role>CEO</role>
</Employee>
<Employee id="3">
<age>40</age>
<name>Tom</name>
<gender>Male</gender>
<role>Manager</role>
</Employee>
<Employees>
I want this to be so.
<?xml version="1.0" encoding="UTF-8"?>
<Employees>
<Employee id="1">
<age>29</age>
<name>Pankaj</name>
<gender>Male</gender>
<role>Java Developer</role>
</Employee>
<Employee id="2">
<age>35</age>
<name>Lisa</name>
<gender>Female</gender>
<role>CEO</role>
</Employee>
<Employee id="3">
<age>40</age>
<name>Tom</name>
<gender>Male</gender>
<role>Manager</role>
</Employee>
<Employees>
<?xml version="1.0" encoding="UTF-8"?>
<Employees>
<Employee id="1">
<age>29</age>
<name>Pankaj</name>
<gender>Male</gender>
<role>Java Developer</role>
</Employee>
<Employee id="2">
<age>35</age>
<name>Lisa</name>
<gender>Female</gender>
<role>CEO</role>
</Employee>
<Employee id="3">
<age>40</age>
<name>Tom</name>
<gender>Male</gender>
<role>Manager</role>
</Employee>
<Employees>