How does conda work inside?

I searched for a while but did not find a satisfactory answer:

How does conda ( http://conda.pydata.org ) work? Any details are welcome ...

Also, since it is python agnostic and seems to work so well and smoothly, why isn't it used as a general-purpose package manager like apt or yum?

What are the limitations of using only conda as a package manager? Will this work?

Or vice versa, why? can't apt and yum provide conda functionality? Is Konda better than a package manager or just another?

Thanks for any tips!

+6
source share
3 answers

I explain this a lot in my SciPy 2014 conversations. Let me give you a little sketch here.

Firstly, the conda package is really simple. These are just tarball files that need to be installed along with some metadata in the info directory. For example, the conda package for python is a file archive

 info/ files index.json ... bin/ python ... lib/ libpython.so python2.7/ ... ... ... 

You can see exactly how it looks by looking at the extracted packages in the Anaconda pkgs . The full specification is at http://conda.pydata.org/docs/spec.html .

When conda installs this, it extracts the tarball into the pkgs directory and hard links the files into the installation environment. Finally, some files that have some hard-coded installation paths are replaced (usually these are shebang strings).

This is basically it. There is something else that happens in terms of resolving dependencies, but as soon as he knows what packages he is going to install, how he does it.

The process of creating a package is a bit more complicated. @mattexx's answer and the document that it links describes a bit of the canonical way to build a package using the conda assembly.

To answer other questions:

Also, since it is python agnostic and seems to work so well and smoothly, why isn't it used as a general-purpose package manager like apt or yum?

Of course you can. The only thing that limits this is the set of packages that were created for conda. On Windows, this is a very good option, since there are no system package managers like Linux.

What are the limitations of using only conda as a package manager? Will this work?

This will work if you have conda packages for everything that interests you. The main limitation is that conda only wants to install things in the conda environment, so anything that requires specific installation locations on the system may not be very suitable for conda (although it is still possible if you set this location as the path medium). Or, for example, conda may not be a suitable replacement for project-level package managers, such as a gazebo.

In addition, conda should probably not be used to manage system-level libraries (libraries that must be installed in the / prefix), such as kernel extensions or the kernel itself, unless you intend to create a distribution that uses conda as a package manager explicitly .

The main thing I'm saying is that conda packages are usually rearranged, which means that the package installation prefix doesn't matter. This is why hard-coded paths change, for example, as part of the installation process. This also means that dynamic libraries built using the conda assembly will have their own RPATH (on Linux), and installation names (on OS X) will automatically change to use relative paths instead of absolute ones.

Or vice versa, why? can't apt and yum provide conda functionality? Is Konda better than a package manager or just another?

In a way, it’s better, but in a way it’s not. Your system package manager knows your system, and there are packages that won't be in the cond (and some, like the kernel, probably shouldn't be in the cond).

The main advantage of conda is the concept of the environment. Since packages can be moved, you can install the same package in several places and effectively have completely independent installations of everything, mostly for free.

Does he use some kind of container packaging

No, the only "containerization" has separate installation directories and does the movement of packages.

or static linking of all dependencies,

Dependency binding is completely dependent on the package itself. Some packages statically link their dependencies, some do not. Dynamically linked libraries have their own loading paths, as I described above, to be roaming.

why is it so cross-platform?

"Cross-platform" in this case means "cross-operating system." Although the same binary package cannot work on OS X, Linux, and Windows, the fact is that the conda itself works the same on all three, so if you have the same packages created for all three platforms, you can manage them anyway no matter where you are on.

+13
source

I am not a software specialist, but I have been using conda to support internal storage for several months, so I can share an understanding of the "advanced user". There are many questions here, so I will try to answer them in order.

How does conda ( http://conda.pydata.org ) work internally? Any details are welcome ...

The shortest link I can provide is the conda-build file , which explains in detail the conda recipes.

TL DR Recipes are folders with the meta.yaml configuration file that describes the package in terms of name, version, source location, dependencies (build, test, run) and basic tests to run after installation. It also contains a build script (s) ( build.sh and / or bld.bat for linux and win, respectively) that perform any build steps other than loading the source.

Installation consists (in short) of loading the source, creating the build environment, creating, creating the test environment, and testing. You can install something system-wide or install it in the environment:

 conda install -n myenv mypkg # install only in myenv conda install mypkg # install globally 

Activating the environment works just like with virtualenv:

 source activate myenv 

What are the limitations of using only conda as a package manager? Will this work?

It will work. You can install whatever you want with conda if you have a recipe that supports your environment. The problem you will encounter is package support. Conda developers and users have created an ecosystem of packages across multiple channels, but binary package support is pretty much limited to what Python packages typically need, and many of them are supported on only one or two platforms. apt, yum, etc. users support all kinds of materials for their respective platforms.

In our case, we need to support Ubuntu and OSX, so we support many platform-dependent binary packages through puppets and other stupid witchcraft, and we use conda to support Python packages for both platforms. If conda packages existed for all the binary packages we use, I could use conda instead of apt, brew, etc., but I would risk getting a significant recipe service if the recipes we used were out of date. This is good for us in the case of Python package management, where conda fills a huge void, but I am not ready to take this for packages that we have existing support tools for. We will see if my thinking changes as the Konda ecosystem ripens. One tool to manage them all would be fine, but I don’t think Konda is ready for me to take this leap.

Does it use some kind of containerization or static linking of all dependencies, why is it so cross-platform?

Cross-platform can have many meanings. For Python packages, cross-platform means that you can create environments with any version of python and packages you need. For Linux / win flavors and distributions, you can do as much as you like in your build script based on the environment. As an example, consider a conda build script for qt . It has appropriate settings for OSX and Linux. The script can do whatever it wants. You can switch to the OS version or whatever. Many recipes simply fail if they do not support the installation platform.

Hope you found this helpful.

+3
source

I see that no one who really understands how conda works is ready to share their knowledge. It is sad...

I can suggest a high level sequence of conda build actions:

  • Look at meta.yaml to find RUN + BUILD excerpts.
  • Creates a new env called: '_build'
    • Installs RUN + BUILD dependencies on meta.yaml
  • Print source code on conda_bld / work
  • Creates a package:
    • take snapshot1 full environment
    • home_dir: 'conda_bld / work', run: 'sh build.sh' 'setup.py install conda_bld / work' (installed locally in _build env)
    • take snapshot2 full environment
    • Package Content: "diff snapshot1 snapshot2"
  • Runs tests:
    • Creates '_test' ENV with '{package just created, RUN DEPS}'
    • runs tests

This is via the @asmeurer youtube link on the side to get you started.

0
source

Source: https://habr.com/ru/post/980460/


All Articles