Reading IDX file type in Java

Question

I created an image classifier in Java that I would like to test on the images presented here: http://yann.lecun.com/exdb/mnist/

Unfortunately, if you download train-images-idx3-ubyte.gz or any of the 3 files, they all have a file type: .idx1-ubyte

First question: I was wondering if someone could give me instructions on how to make .idx1-ubyte into raster image files (.bmp)?

Second question: Or just how can I read these files at all?

IDX file format information: IDX file format is a simple format for vectors and multidimensional matrices of various numeric types. Main format:

magic number size in dimension 0 size in dimension 1 size in dimension 2 ..... size in dimension N data 

The magic number is an integer (first MSB). The first 2 bytes are always 0.

The third byte encodes the data type:

 0x08: unsigned byte 0x09: signed byte 0x0B: short (2 bytes) 0x0C: int (4 bytes) 0x0D: float (4 bytes) 0x0E: double (8 bytes) 

The 4th byte encodes the number of dimensions of the vector / matrix: 1 for vectors, 2 for matrices ....

The dimensions in each dimension are 4-byte integers (first MSB, high endian, as on most processors other than Intel).

Data is stored as in array C, i.e. the index of the last measurement changes the fastest.

Answer 1

Pretty Straightforward, as WPrecht said: "The URL describes the format you must decode." This is my ImageSet exporter for idx file, not very clean, but does what it should do.

 public class IdxReader { public static void main(String[] args) { // TODO Auto-generated method stub FileInputStream inImage = null; FileInputStream inLabel = null; String inputImagePath = "CBIR_Project/imagesRaw/MNIST/train-images-idx3-ubyte"; String inputLabelPath = "CBIR_Project/imagesRaw/MNIST/train-labels-idx1-ubyte"; String outputPath = "CBIR_Project/images/MNIST_Database_ARGB/"; int[] hashMap = new int[10]; try { inImage = new FileInputStream(inputImagePath); inLabel = new FileInputStream(inputLabelPath); int magicNumberImages = (inImage.read() << 24) | (inImage.read() << 16) | (inImage.read() << 8) | (inImage.read()); int numberOfImages = (inImage.read() << 24) | (inImage.read() << 16) | (inImage.read() << 8) | (inImage.read()); int numberOfRows = (inImage.read() << 24) | (inImage.read() << 16) | (inImage.read() << 8) | (inImage.read()); int numberOfColumns = (inImage.read() << 24) | (inImage.read() << 16) | (inImage.read() << 8) | (inImage.read()); int magicNumberLabels = (inLabel.read() << 24) | (inLabel.read() << 16) | (inLabel.read() << 8) | (inLabel.read()); int numberOfLabels = (inLabel.read() << 24) | (inLabel.read() << 16) | (inLabel.read() << 8) | (inLabel.read()); BufferedImage image = new BufferedImage(numberOfColumns, numberOfRows, BufferedImage.TYPE_INT_ARGB); int numberOfPixels = numberOfRows * numberOfColumns; int[] imgPixels = new int[numberOfPixels]; for(int i = 0; i < numberOfImages; i++) { if(i % 100 == 0) {System.out.println("Number of images extracted: " + i);} for(int p = 0; p < numberOfPixels; p++) { int gray = 255 - inImage.read(); imgPixels[p] = 0xFF000000 | (gray<<16) | (gray<<8) | gray; } image.setRGB(0, 0, numberOfColumns, numberOfRows, imgPixels, 0, numberOfColumns); int label = inLabel.read(); hashMap[label]++; File outputfile = new File(outputPath + label + "_0" + hashMap[label] + ".png"); ImageIO.write(image, "png", outputfile); } } catch (FileNotFoundException e) { // TODO Auto-generated catch block e.printStackTrace(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } finally { if (inImage != null) { try { inImage.close(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } } if (inLabel != null) { try { inLabel.close(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } } } 

Answer 2

The url describes the format you have to decode, and they mention that it is non-standard, so an obvious Google search does not cause any usage code. However, it is very straightforward with a heading followed by a 28x28 pixel matrix with grayscale values ​​of 0-255.

After you read the data (do not forget to pay attention to the byte order), creating BMP files is very simple.

I recommend you the following article:

How to make a BMP image from a byte array of pixels in Java

their question is about color, but their code is already working for shades of gray, and that anyway you need, you can get something from this snippet.

Answer 3

I created several classes for reading the MNIST handwritten numeric set using Java. Classes can read files after they have been unpacked (unzipped) from files available on the download site. Classes that allow you to read original (compressed) files are part of the small MnistReader project.

These following classes are self-contained (this means that they have no dependencies on third-party libraries) and are mainly located in the Public Domain - this means that they can simply be copied to their own projects. (Attributes are welcome, but not required):

MnistDecompressedReader Class:

 import java.io.DataInputStream; import java.io.FileInputStream; import java.io.IOException; import java.io.InputStream; import java.nio.file.Path; import java.util.Objects; import java.util.function.Consumer; /** * A class for reading the MNIST data set from the <b>decompressed</b> * (unzipped) files that are published at * <a href="http://yann.lecun.com/exdb/mnist/"> * http://yann.lecun.com/exdb/mnist/</a>. */ public class MnistDecompressedReader { /** * Default constructor */ public MnistDecompressedReader() { // Default constructor } /** * Read the MNIST training data from the given directory. The data is * assumed to be located in files with their default names, * <b>decompressed</b> from the original files: * extension) : * <code>train-images.idx3-ubyte</code> and * <code>train-labels.idx1-ubyte</code>. * * @param inputDirectoryPath The input directory * @param consumer The consumer that will receive the resulting * {@link MnistEntry} instances * @throws IOException If an IO error occurs */ public void readDecompressedTraining(Path inputDirectoryPath, Consumer<? super MnistEntry> consumer) throws IOException { String trainImagesFileName = "train-images.idx3-ubyte"; String trainLabelsFileName = "train-labels.idx1-ubyte"; Path imagesFilePath = inputDirectoryPath.resolve(trainImagesFileName); Path labelsFilePath = inputDirectoryPath.resolve(trainLabelsFileName); readDecompressed(imagesFilePath, labelsFilePath, consumer); } /** * Read the MNIST training data from the given directory. The data is * assumed to be located in files with their default names, * <b>decompressed</b> from the original files: * extension) : * <code>t10k-images.idx3-ubyte</code> and * <code>t10k-labels.idx1-ubyte</code>. * * @param inputDirectoryPath The input directory * @param consumer The consumer that will receive the resulting * {@link MnistEntry} instances * @throws IOException If an IO error occurs */ public void readDecompressedTesting(Path inputDirectoryPath, Consumer<? super MnistEntry> consumer) throws IOException { String testImagesFileName = "t10k-images.idx3-ubyte"; String testLabelsFileName = "t10k-labels.idx1-ubyte"; Path imagesFilePath = inputDirectoryPath.resolve(testImagesFileName); Path labelsFilePath = inputDirectoryPath.resolve(testLabelsFileName); readDecompressed(imagesFilePath, labelsFilePath, consumer); } /** * Read the MNIST data from the specified (decompressed) files. * * @param imagesFilePath The path of the images file * @param labelsFilePath The path of the labels file * @param consumer The consumer that will receive the resulting * {@link MnistEntry} instances * @throws IOException If an IO error occurs */ public void readDecompressed(Path imagesFilePath, Path labelsFilePath, Consumer<? super MnistEntry> consumer) throws IOException { try (InputStream decompressedImagesInputStream = new FileInputStream(imagesFilePath.toFile()); InputStream decompressedLabelsInputStream = new FileInputStream(labelsFilePath.toFile())) { readDecompressed( decompressedImagesInputStream, decompressedLabelsInputStream, consumer); } } /** * Read the MNIST data from the given (decompressed) input streams. * The caller is responsible for closing the given streams. * * @param decompressedImagesInputStream The decompressed input stream * containing the image data * @param decompressedLabelsInputStream The decompressed input stream * containing the label data * @param consumer The consumer that will receive the resulting * {@link MnistEntry} instances * @throws IOException If an IO error occurs */ public void readDecompressed( InputStream decompressedImagesInputStream, InputStream decompressedLabelsInputStream, Consumer<? super MnistEntry> consumer) throws IOException { Objects.requireNonNull(consumer, "The consumer may not be null"); DataInputStream imagesDataInputStream = new DataInputStream(decompressedImagesInputStream); DataInputStream labelsDataInputStream = new DataInputStream(decompressedLabelsInputStream); int magicImages = imagesDataInputStream.readInt(); if (magicImages != 0x803) { throw new IOException("Expected magic header of 0x803 " + "for images, but found " + magicImages); } int magicLabels = labelsDataInputStream.readInt(); if (magicLabels != 0x801) { throw new IOException("Expected magic header of 0x801 " + "for labels, but found " + magicLabels); } int numberOfImages = imagesDataInputStream.readInt(); int numberOfLabels = labelsDataInputStream.readInt(); if (numberOfImages != numberOfLabels) { throw new IOException("Found " + numberOfImages + " images but " + numberOfLabels + " labels"); } int numRows = imagesDataInputStream.readInt(); int numCols = imagesDataInputStream.readInt(); for (int n = 0; n < numberOfImages; n++) { byte label = labelsDataInputStream.readByte(); byte imageData[] = new byte[numRows * numCols]; read(imagesDataInputStream, imageData); MnistEntry mnistEntry = new MnistEntry( n, label, numRows, numCols, imageData); consumer.accept(mnistEntry); } } /** * Read bytes from the given input stream, filling the given array * * @param inputStream The input stream * @param data The array to be filled * @throws IOException If the input stream does not contain enough bytes * to fill the array, or any other IO error occurs */ private static void read(InputStream inputStream, byte data[]) throws IOException { int offset = 0; while (true) { int read = inputStream.read( data, offset, data.length - offset); if (read < 0) { break; } offset += read; if (offset == data.length) { return; } } throw new IOException("Tried to read " + data.length + " bytes, but only found " + offset); } } 

MnistEntry Class:

 import java.awt.image.BufferedImage; import java.awt.image.DataBuffer; import java.awt.image.DataBufferByte; /** * An entry of the MNIST data set. Instances of this class will be passed * to the consumer that is given to the {@link MnistCompressedReader} and * {@link MnistDecompressedReader} reading methods. */ public class MnistEntry { /** * The index of the entry */ private final int index; /** * The class label of the entry */ private final byte label; /** * The number of rows of the image data */ private final int numRows; /** * The number of columns of the image data */ private final int numCols; /** * The image data */ private final byte[] imageData; /** * Default constructor * * @param index The index * @param label The label * @param numRows The number of rows * @param numCols The number of columns * @param imageData The image data */ MnistEntry(int index, byte label, int numRows, int numCols, byte[] imageData) { this.index = index; this.label = label; this.numRows = numRows; this.numCols = numCols; this.imageData = imageData; } /** * Returns the index of the entry * * @return The index */ public int getIndex() { return index; } /** * Returns the class label of the entry. This is a value in [0,9], * indicating which digit is shown in the entry * * @return The class label */ public byte getLabel() { return label; } /** * Returns the number of rows of the image data. * This will usually be 28. * * @return The number of rows */ public int getNumRows() { return numRows; } /** * Returns the number of columns of the image data. * This will usually be 28. * * @return The number of columns */ public int getNumCols() { return numCols; } /** * Returns a <i>reference</i> to the image data. This will be an array * of length <code>numRows * numCols</code>, containing values * in [0,255] indicating the brightness of the pixels. * * @return The image data */ public byte[] getImageData() { return imageData; } /** * Creates a new buffered image from the image data that is stored * in this entry. * * @return The image */ public BufferedImage createImage() { BufferedImage image = new BufferedImage(getNumCols(), getNumRows(), BufferedImage.TYPE_BYTE_GRAY); DataBuffer dataBuffer = image.getRaster().getDataBuffer(); DataBufferByte dataBufferByte = (DataBufferByte) dataBuffer; byte data[] = dataBufferByte.getData(); System.arraycopy(getImageData(), 0, data, 0, data.length); return image; } @Override public String toString() { String indexString = String.format("%05d", index); return "MnistEntry[" + "index=" + indexString + "," + "label=" + label + "]"; } } 

The reader can be used to read uncompressed files. The result will be instances of MnistEntry that are passed to the consumer:

 MnistDecompressedReader mnistReader = new MnistDecompressedReader(); mnistReader.readDecompressedTraining(Paths.get("./data"), mnistEntry -> { System.out.println("Read entry " + mnistEntry); BufferedImage image = mnistEntry.createImage(); ... }); 

The MnistReader project contains several examples of how these classes can be used to read compressed or uncompressed data or to generate PNG images from MNIST records.