Strange Puzzle - Invalid memory access for location

Question

The error message that I get sequentially:

Invalid memory access for location 0x8 rip = 0x10cf4ab28

What I am doing is making a basic backtesting verification system that iterates huge arrays of stocks / historical data using various algorithms using java + eclipse on the latest Mac OS X.

I tracked down the code that seems to be calling it. The method that is used to get massive data arrays is called thousands of times. Nothing has been preserved, so I don’t think there is a memory leak. However, it seems that the set limit is about 7000 times, I can iterate over it before I get a memory error.

It is strange that it works fine in debug mode. Does anyone know which debug mode works differently in Eclipse?

Providing jvm with more memory doesn't help, and it seems to work just fine using -xint. And again, it works fine in debug mode.

public static List<Stock> getStockArray(ExchangeType e){ List<Stock> stockArray = new ArrayList<Stock>(); if(e == ExchangeType.ALL){ stockArray.addAll(getStockArray(ExchangeType.NYSE)); stockArray.addAll(getStockArray(ExchangeType.NASDAQ)); }else if(e == ExchangeType.ETF){ stockArray.addAll(etfStockArray); }else if(e == ExchangeType.NYSE){ stockArray.addAll(nyseStockArray); }else if(e == ExchangeType.NASDAQ){ stockArray.addAll(nasdaqStockArray); } return stockArray; } 

A simple loop like this, repeated more than 1000 times, will cause a memory error. But not in debug mode.

 for (Stock stock : StockDatabase.getStockArray(ExchangeType.ETF)) { System.out.println(stock.symbol); } 

Answer 1

This is probably too late, but for other people who have this problem, you will fix it. Instead of using the extended for loop, use the regular one. I'm serious.

Example:

 for(int stock = 0; stocks < StockDataBase.getStockArray(ExchangeType.ETF).size(); stock++) 

You can take it from there.

This is because you create new objects every time improved for cycle loops, but not in a regular loop. The garbage collector probably does not have enough time to simultaneously address all of these new objects.

Answer 2

Rob, your code can be much more efficient and use a lot less memory if you use a lot of suggestions in the comments ... which will naturally fix your original problem.

 public enum ExchangeType { ALL, ETF, NYSE, NASDAQ; private static EnumMap<ExchangeType, List<Stock> stocks; // Do this during your initialization routine, or if something changes public static loadStock(ExchangeType et, List<stock> stockList) { stocks.put(et, stockList); } public List<Stock> getStocks() { return stocks.get(this); } } 

Then, for security, you can even implement your own ImmutableList ...

 public class ImmutableList<E> implements List<E> { private ArrayList<E> _internal; // Do something like this for modification methods public boolean add(E e) { throw new UnsupportedOperationException("This list is immutable!"); } // Do something like this for access methods public E get(int index) { return _internal.get(index); } } 

This should control your memory and significantly improve performance.

In any case, if this does not fix your problem, as many others said, this is probably a JVM error. As you noticed, compilation (i.e. -XX:CompileThreshold=100000 ) is a quick and dirty fix that should solve your problem at least for now ...

Answer 3

Not the best solution, but you say that you are not modifying the lists, so you can do something like this:

 public static List<Stock> getStockArray(ExchangeType e){ if(e == ExchangeType.ALL){ List<Stock> stockArray = new ArrayList<Stock>(nyseStockArray.size() + nasdaqStockArray.size()); stockArray.addAll(nyseStockArray); stockArray.addAll(nasdaqStockArray); return stockArray; }else if(e == ExchangeType.ETF){ return etfStockArray; }else if(e == ExchangeType.NYSE){ return nyseStockArray; }else if(e == ExchangeType.NASDAQ){ return nasdaqStockArray; } else { return new ArrayList<Stock>(); }