"sujust all" error

#1

Hello,
I beginning just to use Clip os. I must to install Clip os on a Centos7 distrib. Install si ok and i update the path. Just --list responds correctly.
But when i try to do a “sujust all”, i have the following error:
just products/clipos/all
just core/all
just …/sdk/all
[X] Uncaught unknown exception ReadError:
file could not be opened successfully
error: Recipe bootstrap failed with exit code 1
error: Recipe sdk failed on line 20 with exit code 1
error: Recipe core failed on line 13 with exit code 1
error: Recipe clipos failed on line 12 with exit code 1

I have searchd for a solution but i do not progess.
Could you help me?

thanks
alain

#2

Hello Alain,

Thanks for taking the time to report your issue. However, we are not really sure to identify the source of the issue you are encountering with the log output you sent.
Also please keep in mind that CentOS is not part of the list of Linux distributions we “officially” support for the CLIP OS toolkit (but Fedora 29 is one of them, see the post-scriptum below for the list of packages we install on Fedora 29).

Could you please try to reproduce the following commands and report any issue you encounter with them:

  1. The following commands are intended to be run from the location of the CLIP OS source tree root (i.e. where you did the repo init command) and from a fresh shell invite (i.e. not in a previously CLIP OS toolkit-activated shell).

  2. Please make sure to have a CLIP OS source tree up-to-date (we merged some important changes yesterday, and I would like to make sure that you are not encountering a bug with something we merged yesterday).
    Please note also that this will “forget” any change you might have brought in any repository of the CLIP OS source tree (this is if you changed something, obviously):

    $ repo sync -j5 -d
    
  3. Ensure to recreate from scratch the CLIP OS toolkit virtual environment:

    $ toolkit/setup.sh
    $ source toolkit/activate
    (toolkit) $
    

    Please notice that your shell invite MUST be now prefixed with (toolkit) to indicate that your $PATH has been properly overrided with our tools location. If not, please report it to us.

    Side note: If toolkit/setup.sh happens to fail, you can retry the above commands after having manually deleted the whole directory run/venv.

  4. Retry the all-in-one command that:

    (toolkit) $ sujust all
    

If after all of this you still encounter the exact same result, please paste here the return of the following command:

$ cosmk --debug bootstrap clipos/sdk

Regards,

– Nicolas


PS: FWIW, here is the current list of installed packages we use on our CI systems that use Fedora 29 as base distribution for building CLIP OS:

dnf install \
    python2 python3-devel \
    gnupg git git-lfs openssh-clients \
    @development-tools \
    runc sudo squashfs-tools \
    bash diffutils \
    qemu libvirt-devel libvirt-daemon \
    rust cargo \
    dumb-init \
    bsdtar
#3

Nicolas,
About fedora 29 packages i do not found the package dumb-init which is used for Centos7. I looking for it. ( it’s ok for all others)

About the issue, after manipulations described in your previous mail, I have the same behavior, and the same message. see hereafter the result of your command “cosmk --debug bootstrap clipos/sdk”
[.] Parsed command line arguments: Namespace(debug=True, no_sudo=False, recipe=‘clipos/sdk’, subcommand=‘bootstrap’)
[.] euid != 0, recalling ourselves through sudo.
[.] Parsed command line arguments: Namespace(debug=True, no_sudo=False, recipe=‘clipos/sdk’, subcommand=‘bootstrap’)
[.] registering in advance output nodes ownership fix at exit
[.] Parsing ‘/home/alain/clipos/products/clipos/properties.toml’…
[.] Parsing “/home/alain/clipos/products/clipos/sdk/recipe.toml”…
[X] Uncaught unknown exception ReadError:
file could not be opened successfully
Traceback (most recent call last):
File “/home/alain/clipos/toolkit/clipostoolkit/cosmk/main.py”, line 299, in main
getattr(recipe, args.subcommand)(**subcommand_kwargs)
File “/home/alain/clipos/toolkit/clipostoolkit/cosmk/features.py”, line 386, in bootstrap
self.recipe.config[“bootstrap”][“env”].items()
File “/home/alain/clipos/toolkit/clipostoolkit/cosmk/sdk.py”, line 210, in bootstrap
tarfile.open(rootfs_archive, “r:*”) as tar:
File “/usr/local/lib/python3.7/tarfile.py”, line 1576, in open
raise ReadError(“file could not be opened successfully”)
tarfile.ReadError: file could not be opened successfully

[.] Fixing ownership nodes in output (“out/”) and cache (“cache/”) directories…

Thanks
alain

#4

Oh sorry, this package is absolutely unnecessary for the CLIP OS toolkit. I just forgot to strip it from the list I gave you, you do not need to install it on your machine.
(dumb-init is just a dependency for our CI builder environments which are containerized (this is a simple signals forwarder for the containerized child processes).)


OK, it looks like you may have corrupted files in the Git LFS-managed repositories (at least in assets/gentoo).
Could you please run this command to check that no repository has corrupted Git LFS objects:

repo forall -g lfs -c 'git lfs fsck'

If the output does not look like a series of messages telling that Git LFS fsck OK, then you can proceed with the following commands to fix your repositories:

repo forall -g lfs -c 'git lfs pull && git checkout .'

If the Git LFS fsck step above succeeded, then could you please confirm that this command exits successfully without any error message on stderr:

tar -tvf 'assets/gentoo/stage3-amd64-hardened+nomultilib-20190224T214502Z.tar.xz' >/dev/null
1 Like
#5

Hello,
i lunche your command : repo forall -g lfs -c ‘git lfs fsck’
it’s ok but i have a warning about version of python
warning: Python 3 support is currently experimental. YMMV.
Please use Python 2.7 instead.
/home/alain/clipos/.repo/repo/main.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module’s documentation for alternative uses
import imp
Git LFS fsck OK
Git LFS fsck OK
Git LFS fsck OK
Git LFS fsck OK
Git LFS fsck OK
so I launch the appropriate command: tar -tvf…
I do not hav any message, no error…
thanks
alain

#6

This comes from the fact that you are using repo with Python 3 rather than Python 2.7. However, repo is not fully supported with Python 3 yet.
But your issue does not seem to come from that. I think you can safely ignore this.

OK. How did you get Python 3.7 for CentOS 7? Did you compile and install it manually? Or did you install it through a third-party package?

What happens if you try to do the following command?

(toolkit) $ python3 -m tarfile -v -l 'assets/gentoo/stage3-amd64-hardened+nomultilib-20190224T214502Z.tar.xz'

Does it succeed to list the contents of the Gentoo stage3 archive without any error?

#7

about python 3.7 i have installed it with wget ./configure, make altinstall.
When i launch your `python3 -m tarfile…’ command
I have the hereafter error message:
‘assets/gentoo/stage3-amd64-hardened+nomultilib-20190224T214502Z.tar.xz’ is not a tar archive.

thanks
alain

#8

OK, then it seems that you have compiled Python 3.7 without LZMA support (maybe due to the fact that you may have been missing liblzma headers on your machine during compilation).
This would explain the fact that the tarfile Python module cannot extract the XZ compressed part of this .tar.xz file and thus cannot interpret this file as a tar archive.

Could you please use another method to install a fully-featured version of the Python 3.6+ interpreter?

Please also note that Python 3.6 is also supported by the CLIP OS toolkit and seems to be provided by Software Collections.

#9

hello,
I have installed the pylzma lib in the python3 install of centos (/usr/…). and i can launch your command without issue. The file is ‘untared’
but… it is out of the clipos sdk
I activate sdk, clipos is in the prompt, “just list” is ok, python version is 3.7. the error is again present and your command not runs correctly. I install the lzma lib to try (pip install pylzma). the install runs correctly ( pylzma is not already installed (!) and pip installs it). Has Clipos SDK its own python3 install? I relaunch your command and the error is again there.
thanks
alain

#10

I’ve re installed fron scratch centos and python 3, using softwarecollections.org. but now, it’s toolkit/setup.sh do not run, command “python3 -Im ensurepip --upgrade --default-pip” returns non-zero exit status 1.

#11

What I meant by “LZMA support” is not about the pylzma Python module but about the support of LZMA by the Python standard library, i.e. the lzma module.
pylzma is just a third-party implementation of LZMA that were particularly useful for early versions of Python (or for advanced LZMA implementation handling) that we deliberately do not use since Python supports LZMA in its standard library (lzma module) since the version 3.3.

You should not extract that file manually. It is up to cosmk to do it appropriately in one of its internal temporary directory where it will work.

…because cosmk makes use of lzma module through the use of the tarfile module (also part of the standard library) and not pylzma. :frowning_face:

The CLIP OS toolkit only makes use of the Python 3.6+ interpreter (and the standard library that comes along) and deliberately ignore all the Python packages you may install on your system via your distro’s package manager or via pip outside the CLIP OS toolkit virtualenv.
Actually, this is precisely the purpose of the Python virtualenv: we use this in order to avoid any potential Python packages version discrepancy between developers (due to the fact that they do not use the same Linux distro) by enforcing our third-party (and vendored) Python packages “ecosystem”.

Therefore to answer your question, the only requirements about Python for the CLIP OS build environment are to have:

  1. a fully-featured Python 3.6+ interpreter with its standard library on your machine (i.e. what should be provided by your Linux distro with a usually named “python3” package),
  2. and the appropriate CPython headers to compile packages embedding CPython code that we may provide in order to link properly against your Python 3.6+ interpreter (i.e. what is provided by a usually named “python3-dev” package).

Are you sure to have entered the rh-python36 environment with the command scl enable rh-python36 bash?
If so, I am afraid that we cannot help you further with this as we do not support CentOS 7 for the build environment and we do not plan to do it at this moment.

We invite you to use one of the four Linux distributions we actively support, that are:

  • Debian sid (aka. unstable)
  • Ubuntu LTS (18.04 by this date), but version 18.10 and 19.04 should also work
  • Fedora stable (29 by this date), but version 28 should also work
  • Arch Linux (“rolling-release” distro)

You can find more details about the build environment requirements in the documentation.

#12

Ok I understand.
I have tested the Python installation, That’s runs correctly.
I will try on fedora for the moment
Thanks a lot about the time past to help me.
alain