Package Details: python-safetensors 0.5.3-1

Git Clone URL: https://aur.archlinux.org/python-safetensors.git (read-only, click to copy)
Package Base: python-safetensors
Description: Simple, safe way to store and distribute tensors
Upstream URL: https://github.com/huggingface/safetensors
Licenses: Apache-2.0
Submitter: Henry-ZHR
Maintainer: envolution
Last Packager: envolution
Votes: 7
Popularity: 0.34
First Submitted: 2023-01-08 09:54 (UTC)
Last Updated: 2025-03-11 02:41 (UTC)

Dependencies (28)

Sources (1)

Latest Comments

« First ‹ Previous 1 2 3 4 Next › Last »

mane.andrea commented on 2023-12-09 03:42 (UTC)

Sorry for the delay. I overlooked the notification. Will work on an updated PKGBUILD asap

daskol commented on 2023-12-09 02:41 (UTC)

In my optinion, the best solution is not put any .gitignore with glob pattern in AUR repo since there is no actual need in blacklisting files at all.

@h3ss I believe that it'd be better to create an empty directory .git at $srcdir/safetensors-$pkgver rather than patching or sed'ing (see ignore::WalkBuilder). Anyway, we should create an issue in PyO3/maturin. It is stupid that it traverses out of repo root. What about submodules? Why I should tweak include/ignore filters in pyproject.toml? Weird.

diff --git a/PKGBUILD b/PKGBUILD
index cb94dc8..4d04fcf 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -38,6 +38,10 @@ checkdepends=('python-pytorch'
 source=("${pkgname}-${pkgver}.tar.gz::${url}/archive/refs/tags/v${pkgver}.tar.gz")
 sha512sums=('da38f184f2c0730a59b1d4ca490758b71cb5ed3e1e16741bbf4b7bfac35fa1a4f24e5acbc35f3b38f10e4c46d81554896fbd95accfd671ee066c9fcbd50551ab')

+prepare() {
+  mkdir "safetensors-${pkgver}/.git"
+}
+
 build() {
   cd "safetensors-${pkgver}/bindings/python"
   python -m build --wheel --no-isolation

@xiota There is the same issue with python-tokenziers.

h3ss commented on 2023-12-05 03:34 (UTC)

The problem is that the AUR package includes a .gitignore file, and the maturin build backend will ignore anything specified in any .gitignore file in the path.

See: - Wheels for mixed Python/Rust project does not include Python source code #885

The contents of the AUR .gitignore are:

**/*
!.SRCINFO
!.gitignore
!PKGBUILD

That first line is the problem, it basically excludes everything from inclusion (except what the Rust part of the build creates apparently).

I have been able to get around this by editing the src/safetensors-0.4.0/bindings/python/pyproject.toml file. Specifically, by adding this under the [tool.maturin] section:

include = ["**/*"]

See: - Maturin User Guide - Configuration - Specifically the include part

When this is done, the resulting .whl file reliably contains the needed Python source files.

So, how can we get around this in the PKGBUILD? By using sed to add the include line to the pyproject.toml in the build() function:

build() {
  cd "safetensors-${pkgver}/bindings/python"
  sed -i 's/\[tool.maturin\]/[tool.maturin]\ninclude = \["**\/*"\]/g' pyproject.toml
  python -m build --wheel --no-isolation
}

After adding that sed line, the package reliably passes all checks and builds.

@mane.andrea Can we get this or something equivalent incorporated into the PKGBUILD? I'm confident this should fix this longstanding issue.

PhotonX commented on 2023-11-26 18:14 (UTC) (edited on 2023-11-26 18:15 (UTC) by PhotonX)

I'm getting the check() errors as well. I don't understand what the underlying problem is, though. One of the errors is, exemlarily:

________________ ERROR collecting tests/test_flax_comparison.py ________________
ImportError while importing test module '/home/photon/.cache/yay/python-safetensors/src/safetensors-0.4.0/bindings/python/tests/test_flax_comparison.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_flax_comparison.py:12: in <module>
    from safetensors import safe_open
E   ImportError: cannot import name 'safe_open' from 'safetensors' (unknown location)

So it searches for safe_open from... itself? I think, I have all the necessary dependencies installed, but none of them provides this...

t3nzor commented on 2023-11-20 02:22 (UTC) (edited on 2023-11-20 02:22 (UTC) by t3nzor)

The built package doesn't seem to install the Python bindings, only the shared library. I was able to work around this as follows:

makepkg -si --nocheck

sudo cp src/safetensors-0.4.0/bindings/python/py_src/safetensors/* /usr/lib/python3.11/site-packages/safetensors

65a commented on 2023-11-18 02:52 (UTC) (edited on 2023-11-18 02:59 (UTC) by 65a)

@phoenix17, I'm glad it works for you, but it doesn't work for me. The result is still failing to import safetensors.torch or crashing in the check() (this depends on which machine I use?!), using a clean git clone of the AUR source then makepkg -si on multiple machines. Here is one failure log: https://pastebin.com/mppuQw0H

phoenix17 commented on 2023-11-16 20:07 (UTC)

In case this helps someone, I was able to build/install this manually via makepkg -si without making any changes to the build script here: https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-safetensors

I can confirm it also worked with python-transformers package on the aur and pytorch (https://archlinux.org/packages/extra/x86_64/python-pytorch/)

pulsarGuy commented on 2023-11-16 04:00 (UTC) (edited on 2023-11-16 04:03 (UTC) by pulsarGuy)

I wound up in @tomsod's situation, and their solution worked for me. Skipping checks did allow the package to be installed, but attempting to import it within python failed. Removing --wheel caused the checks to be successful, the package to be installed, and allowed importing within python. Unfortunately, I do not use torch, so I can't comment on that aspect.

65a commented on 2023-11-14 02:13 (UTC)

This package doesn't properly provide safetensors.torch import. If I follow @tomsod's advice, I can't even import safetensors. I suspect it's something between src and py-src directories, but I don't understand python packaging well enough to understand why this is isn't working (PKGBUILD changes look reasonable, but maybe something new is needed to support an upstream change? idk)

Tomsod commented on 2023-11-08 04:10 (UTC) (edited on 2023-11-08 04:10 (UTC) by Tomsod)

Tests failed for me too, so I skipped them, but the build package didn't work either. Turns out the --wheel option in build() somehow prevents these files from being installed. Without it, everything seems to work fine.

« First ‹ Previous 1 2 3 4 Next › Last »