Extracting images from a .idx3-ubyte or GZIP file via Python

Question

I created a simple function for facerecognition using facerecognizer from OpenCV. It works great with images of people.

Now I would like to do a test using personal characters instead of people. I stumbled upon the MNIST dataset, but they store images in a strange file that I had never seen before.

I just need to extract some images from:

train-images.idx3-ubyte 

and save them in a folder as .gif

Or I do not understand that it is MNIST. If so, where can I get such a dataset?

EDIT

I also have a gzip file:

 train-images-idx3-ubyte.gz 

I am trying to read the contents, but show() not working, and if I read() , I see random characters.

 images = gzip.open("train-images-idx3-ubyte.gz", 'rb') print images.read() 

EDIT

I managed to get a useful conclusion with:

 with gzip.open('train-images-idx3-ubyte.gz','r') as fin: for line in fin: print('got line', line) 

Somehow I need to convert this now into an image, output:

Answer 1

Download training / test images and shortcuts:

• train-images-idx3-ubyte.gz: training image set
• train-tags-idx1-ubyte.gz: training set shortcuts
• t10k-images-idx3-ubyte.gz: test image set
• t10k-tags-idx1-ubyte.gz: test tag set

And unzip them into a working directory, say samples/ .

Get the python-mnist package from PyPi:

 pip install python-mnist 

Import the mnist package and read the tutorial / test images:

 from mnist import MNIST mndata = MNIST('samples') images, labels = mndata.load_training() # or images, labels = mndata.load_testing() 

To display an image on the console:

 index = random.randrange(0, len(images)) # choose an index ;-) print(mndata.display(images[index])) 

You will get something like this:

 ............................ ............................ ............................ ............................ ............................ .................@ @......... ..............@ @@@@......... ............@ @@@............ ..........@ @................ ..........@................. ...........@................ ...........@................ ...........@... @............ ...........@ @@@@ .@.......... ...........@ @@ ...@ @......... ...........@ @ .....@......... ..................@......... ..................@ @........ ..................@ @........ ..................@......... .................@ @......... ...........@..... @.......... ...........@.... @@.......... ............@ @@@............ .............@.............. ............................ ............................ ............................ 

Explanation:

• Each image in a list of images is a Python unsigned list .
• Labels are a Python- array unsigned bytes.

Answer 2

(Using only matplotlib, gzip and numpy)
Extract image data:

 import gzip f = gzip.open('train-images-idx3-ubyte.gz','r') image_size = 28 num_images = 5 import numpy as np f.read(16) buf = f.read(image_size * image_size * num_images) data = np.frombuffer(buf, dtype=np.uint8).astype(np.float32) data = data.reshape(num_images, image_size, image_size, 1) 

Print image:

 import matplotlib.pyplot as plt image = np.asarray(data[2]).squeeze() plt.imshow(image) plt.show() 

Print the first 50 labels:

 f = gzip.open('train-labels-idx1-ubyte.gz','r') f.read(8) for i in range(0,50): buf = f.read(1) labels = np.frombuffer(buf, dtype=np.uint8).astype(np.int64) print(labels) 

Answer 3

Use this to retrieve the mnist database for images and csv shortcuts in python:

https://github.com/sorki/python-mnist

Answer 4

You really can use the idx2numpy package available on PyPI. It is extremely easy to use and directly converts data into arrays. Here is what you should do:

Data loading

Download the MNIST dataset from the official website .
If you are using Linux, you can use wget to get it from the command line. Just run:

 wget http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz wget http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz wget http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz wget http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz 

Data unpacking

Unzip or unzip the data. On Linux you can use gzip

Ultimately, you should have the following files:

 data/train-images-idx3-ubyte data/train-labels-idx1-ubyte data/t10k-images-idx3-ubyte data/t10k-labels-idx1-ubyte 

The data/ prefix is ​​only because I extracted them to a folder called data . Your question looks like you've done everything so far, so keep reading.

Using idx2numpy

Here is simple Python code to read everything from unpacked files as arrays.

 import idx2numpy import numpy as np file = 'data/train-images-idx3-ubyte' arr = idx2numpy.convert_from_file(file) # arr is now a np.ndarray type of object of shape 60000, 28, 28 

Now you can use it with OpenCV Juts in the same way that you display any other image, using something like

 cv.imshow("Image", arr[4]) 

To install idx2numpy, you can use PyPI ( pip package manager). Just run the command:

 pip install idx2numpy