How can I implement a POSIX file descriptor in Python 3?

Question

I would like to write a class that can act like a genuine file descriptor. Its .fileno () method should return a file descriptor that provides all the services expected by the POSIX system.

This is my first foray into POSIX system programming, so I could misunderstand things.

The main motivation is the desire to use a Python object in memory as the stdin or stdout kwarg constructor for the subprocess.Popen constructor, without relying on temporary files or memory mapped files. But I'm not interested in some ingenious trick that will do the job - I really want the Python implementation to be able to answer all the relevant system calls.

Answer 1

You can not. POSIX file descriptors are tracked in the kernel of the operating system, outside the world of Python; you cannot imitate them in python code.

Answer 2

If you want to have a class that can be used as a file when passing to system calls, it must have fileno (), which is the real descriptor of the OS file system. One way to do this without touching the hard drive is to use pipes because they have file descriptors, and system calls can be written to these file descriptors.

I wrote a class that did something using this method for another answer . This is really not what you want to do, but the pipe technique should be doable for you:

 import io import logging import os import select import subprocess import time import threading LOG_FILENAME = 'output.log' logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG) class StreamLogger(io.IOBase): def __init__(self, level): self.level = level self.pipe = os.pipe() self.thread = threading.Thread(target=self._flusher) self.thread.start() def _flusher(self): self._run = True buf = b'' while self._run: for fh in select.select([self.pipe[0]], [], [], 0)[0]: buf += os.read(fh, 1024) while b'\n' in buf: data, buf = buf.split(b'\n', 1) self.write(data.decode()) time.sleep(1) self._run = None def write(self, data): return logging.log(self.level, data) def fileno(self): return self.pipe[1] def close(self): if self._run: self._run = False while self._run is not None: time.sleep(1) os.close(self.pipe[0]) os.close(self.pipe[1]) 

Answer 3

This is my first foray into POSIX system programming, so I could misunderstand things.

Yes.

POSIX file descriptors are just numbers — they are not objects, so you cannot override their methods. For example, 0, 1, and 2 are [usually] valid file descriptors.

"appropriate system calls" are built into the Linux kernel. The Linux kernel maintains a list that maps file descriptors to some internal kernel object (which has methods!), But you cannot insert a new file descriptor from Python. Code running in kernel space is very different from regular ("user mode") code.

Can I suggest you look at subprocess.PIPE, as well as the stdout / stdin / stderr properties or the connect () method for subprocess.Popen objects? This will allow you to start the subprocess, read the data that it outputs, and fully control the data that is sent to it. (I think this is what you are really trying to do ...). If you're interested, then when you play with this, you can look at the source code of subprocess.py to find out how it works.

Here is an example subprocess.PIPE here .

Alternatively, if you really want to implement a complete file system in Python, see FUSE instead of Linking to Python . FUSE includes a C module that runs in the kernel and processes file system requests for a specific directory. It processes them, passing them to a user-space program that can be written in Python. You can open these files from a separate Python program to get a file descriptor for them. This is quite complicated and probably not the best place for beginners.