* Migrate build system to uv
Closes#3787, and is a step towards #3081 and #4022
This change breaks our PyOxidizer bundling process. While we probably
could update it to work with the new venvs & lockfile, my intention
is to use this as a base to try out a uv-based packager/installer.
Some notes about the changes:
- Use uv for python download + venv installation
- Drop python/requirements* in favour of pyproject files / uv.lock
- Bumped to latest Python 3.9 version. The move to 3.13 should be
a fairly trivial change when we're ready.
- Dropped the old write_wheel.py in favour of uv/hatchling. This has
the unfortunate side-effect of dropping leading zeros in our wheels,
which we could try hack around in the future.
- Switch to Qt 6.7 for the dev repo, as it's the first PyQt version
with a Linux/ARM WebEngine wheel.
- Unified our macOS deployment target with minimum required for ARM.
- Dropped unused fluent python files
- Dropped unused python license generation
- Dropped helpers to run under Qt 5, as our wheels were already
requiring Qt 6 to install.
* Build action to create universal uv binary
* Drop some PyOxidizer-related files
* Use Windows ARM64 cargo/node binaries during build
We can't provide ARM64 wheels to users yet due to #4079, but we can
at least speed up the build.
The rustls -> native-tls change on Windows is because ring requires
clang to compile for ARM64, and I figured it's best to keep our Windows
deps consistent. We already built the wheels with native-tls.
* Make libankihelper a universal library
We were shipping a single arch library in a purelib, leading to
breakages when running on a different platform.
* Use Python wheel for mpv/lame on Windows/Mac
This is convenient, but suboptimal on a Mac at the moment. The first
run of mpv will take a number of seconds for security checks to run,
and our mpv code ends up timing out, repeating the process each time.
Our installer stub will need to invoke mpv once first to get it validated.
We could address this by distributing the audio with the installer/stub,
or perhaps by putting the binaries in a .pkg file that's notarized+stapled
and then included in the wheel.
* Add some helper scripts to build a fully-locked wheel
* Initial macOS launcher prototype
* Add a hidden env var to preload our libs and audio helpers on macOS
* qt/bundle -> qt/launcher
- remove more of the old bundling code
- handle app icon
* Fat binary, notarization & dmg
* Publish wheels on testpypi for testing
* Use our Python pin for the launcher too
* Python cleanups
* Extend launcher to other platforms + more
- Switch to Qt 6.8 for repo default, as 6.7 depends on an older
libwebp/tiff which is unavailable on newer installs
- Drop tools/mac-x86, as we no longer need to test against Qt 5
- Add flags to cross compile wheels on Mac and Linux
- Bump glibc target to 2_36, building on Debian Stable
- Increase mpv timeout on macOS to allow for initial gatekeeper checks
- Ship both arm64 and amd64 uv on Linux, with a bash stub to pick
the appropriate arch.
* Fix pylint on Linux
* Fix failure to run from /usr/local/bin
* Remove remaining pyoxidizer refs, and clean up duplicate release folder
* Rust dep updates
- Rust 1.87 for now (1.88 due out in around a week)
- Nom looks involved, so I left it for now
- prost-reflect depends on a new prost version that got yanked
* Python 3.13 + dep updates
Updated protoc binaries + add helper in order to try fix build breakage.
Ended up being due to an AI-generated update to pip-system-certs that
was not reviewed carefully enough:
https://gitlab.com/alelec/pip-system-certs/-/issues/36
The updated mypy/black needed some tweaks to our files.
* Windows compilation fixes
* Automatically run Anki after installing on Windows
* Touch pyproject.toml upon install, so we check for updates
* Update Python deps
- urllib3 for CVE
- pip-system-certs got fixed
- markdown/pytest also updated
This commit explains how to calls a method implemented in a language
from a different language.
This explains how to declare the RPCs, how to call them and how to
implement them. This is based on examples of code at main at the time
of writting. I used permalink to ensure that the links remains
relevant even if the specific examples change later.
The last section is about the special case of calling TypeScript from
Python, which does not use RPC but is still relevant in a bridge
document.
This commit also add a paragraph explaining what protobuf is in the
protobuf documentation, so that new contributors who don't know what
protobuf is can understand why we use it.
* pass --locked to cargo invocation
* update Dockerfile.distroless as well
Co-authored-by: Simon <8466614+SimonBaars@users.noreply.github.com>
---------
Co-authored-by: Simon <8466614+SimonBaars@users.noreply.github.com>
Hardcode them to:
SYNC_PORT=8080
SYNC_BASE=/anki_data
If these env variables are passed into the container with different values,
they are ignored.
The reasons is if the user modifies SYNC_BASE they risk data loss since
anki-sync-server will no longer write data into the volume. If they change
SYNC_PORT they need to also change it when mapping this internal port to the
external port of the container, which could be confusing plus it has no benefit
to allow this since it's always possible to change the external port even if
the internal port is fixed to 8080 (e.g. `-p 1234:8080`).
In both cases there is no benefit to making these values configurable and there
are risks associated.
Unfortunately there is no easy way of implementing this for the
Dockerfile.distroless so it's up to the user not to modify these values.
PUID and PGID are optional env variables to specify the user and group id of
the user that the anki-sync-server process should run with.
This gives more flexibility for solving permission problems with volumes and is
a common pattern for Docker images (e.g. see here:
https://docs.linuxserver.io/general/understanding-puid-and-pgid/)
The anki-sync-server process will write any files with the permissions of the
user it's running with, which can be a problem when you need to access those
files from outside the container or when they are being written into a bind
mount that is owned by a particular user on the host system.
To be able to implement this the entrypoint.sh needs to run as root (since it
needs to create a user and change file permissions). anki-sync-server then
needs to be started with the user 'anki', which is why the new dependency
'su-exec' is required. The user 'anki' and group 'anki-group' can no longer be
created at image build time because then their ids would be fixed.
Also update the build instructions to require building the Docker image inside
the directory where the Dockerfile resides since the build now needs to copy
the entrypoint.sh and it seems wrong the specify the path
docs/syncserver/entrypoint.sh inside the Dockerfile.
Now that an ARM wheel is on PyPI, we no longer need to rely on a
system PyQt to build on ARM. The install is skipped when PYTHONPATH
is set, so older distros with glibc <2.39 can continue to use the
system packages instead.
Otherwise data would be lost by default when removing (or re-creating) a
container.
It would be possible to expose the default directory (e.g.
/home/anki/.syncserver) but it would be different for the two Dockerfiles and
less convenient for users of the Docker container to specify such a long path
when naming their volumes.
Setting the permissions is necessary since anki will be running with 'anki'
user permissions inside the container.
* Qt 6.8.1
Bumps minimum glibc to 2.35, and minimum macOS to 12
* Drop generation of Qt5 packaged build
Closes#3615
* Include qt6 requirements in aqt wheel; drop extra deps
* Fix aqt wheels growing over time
* Add myself to CONTRIBUTORS file
* replace localhost with 127.0.0.1 in syncserver Dockerfile
The healthcheck was failing, presumably because localhost was resolving to ::1
(IPv6), as detailed in this issue: https://github.com/maildev/maildev/pull/500
* docs(docker): Change suggested version numbre
* deps(docker): Bump rust to 1.83.0 and alpine to 3.21.0
* deps(docker): Bump rust to 1.83.0
* CONTRIBUTORS: Add my name
* Add myself to CONTRIBUTORS file
* avoid warning by setting SYNC_PORT as ARG in Dockerfile
1 warning found (use docker --debug to expand):
- UndefinedVar: Usage of undefined variable '$SYNC_PORT'
- rslib(http_server): add `is_running()` method
- rslib(sync): introduce `--healthcheck` argument for health probe in distroless
- doc(syncserver): add table comparing Dockerfile and Dockerfile.distroless
- Expand cross-platform support with distroless
- add `Dockerfile.distroless`
- Dockerfile: bump rust `1.79` to `1.80.1`
- Dockerfile: bump alpine `3.20` to `3.20.2`
Note: Implemented an internal health check because distroless images do not include curl, which is used to reduce image size and attack surface. For more details, see https://blog.sixeyed.com/docker-healthchecks-why-not-to-use-curl-or-iwr/https://github.com/GoogleContainerTools/distroless
fix: failed: check:format:rust
typo
remove extra space
fix failed:check:format:rust
update doc
fetch `host` and `port` using envy
fix: failed: check:format:rust
Update doc + add dockerignore
- dockerignore: This helps avoid sending unwanted files and directories to the builder
- add new line
- I am still experimenting cross platform compilation, I am getting
4.337 From https://github.com/ankitects/rust-url
4.337 * [new ref] bb930b8d089f4d30d7d19c12e54e66191de47b88 -> refs/commit/bb930b8d089f4d30d7d19c12e54e66191de47b88
4.397 error: failed to get `percent-encoding-iri` as a dependency of package `anki v0.0.0 (/app/rslib)`
still checking what could be the issue
fix: failed: check:format:dprint
* Update base images and introduce health endpoint
sync-server: introduce `/health` endpoint to check if the service is reachable.
bump(alpine): bump alpine base image from `3.19` to `3.20`
bump(rust): bump rust-alpine build image from `1.76` to `1.79`
* fix cargo fmt
* add allow clippy::extra_unused_type_parameters
* Remove unused type param (dae)
* Route /health directly (dae)
* Fix for latest axum (dae)
* Simplify the offline build
The two environment variables OFFLINE_BUILD and NO_VENV jointly provide
the ability to build Anki fully offline. This commit boils them down
into just one, namely OFFLINE_BUILD.
The rationale being that first, OFFLINE_BUILD implies the use of
a custom non-networked Python environment.
Second, building Anki with a custom Python environment in a networked
setting is a use case, that we currently do not support.
Developers in need of such a solution may want to give containerized
development environments a try. Users could also look into building
Anki fully offline instead.
* Add documentation for offline builds.
* Add support for offline generation of Sphinx documentation.
Control installation of Sphinx dependencies via the network through the
OFFLINE_BUILD environment variable.
* Add documentation for offline generation of Sphinx documentation.
* Add `extra` directory as a designated ignored folder
Excludes `extra/` from version tracking, file formatters, and file checks.
* Remove pytest cache from exclusion rules
Python test discovery is easy enough to disable for the workspace in VS Code's settings and pytest does not serve any purpose in the context of the project anyway.
* Fix .no-reduce-motion missing from graphs spinner, and not being honored
* Begin migration from protobuf.js -> protobuf-es
Motivation:
- Protobuf-es has a nicer API: messages are represented as classes, and
fields which should exist are not marked as nullable.
- As it uses modules, only the proto messages we actually use get included
in our bundle output. Protobuf.js put everything in a namespace, which
prevented tree-shaking, and made it awkward to access inner messages.
- ./run after touching a proto file drops from about 8s to 6s on my machine. The tradeoff
is slower decoding/encoding (#2043), but that was mainly a concern for the
graphs page, and was unblocked by
37151213cd
Approach/notes:
- We generate the new protobuf-es interface in addition to existing
protobuf.js interface, so we can migrate a module at a time, starting
with the graphs module.
- rslib:proto now generates RPC methods for TS in addition to the Python
interface. The input-arg-unrolling behaviour of the Python generation is
not required here, as we declare the input arg as a PlainMessage<T>, which
marks it as requiring all fields to be provided.
- i64 is represented as bigint in protobuf-es. We were using a patch to
protobuf.js to get it to output Javascript numbers instead of long.js
types, but now that our supported browser versions support bigint, it's
probably worth biting the bullet and migrating to bigint use. Our IDs
fit comfortably within MAX_SAFE_INTEGER, but that may not hold for future
fields we add.
- Oneofs are handled differently in protobuf-es, and are going to need
some refactoring.
Other notable changes:
- Added a --mkdir arg to our build runner, so we can create a dir easily
during the build on Windows.
- Simplified the preference handling code, by wrapping the preferences
in an outer store, instead of a separate store for each individual
preference. This means a change to one preference will trigger a redraw
of all components that depend on the preference store, but the redrawing
is cheap after moving the data processing to Rust, and it makes the code
easier to follow.
- Drop async(Reactive).ts in favour of more explicit handling with await
blocks/updating.
- Renamed add_inputs_to_group() -> add_dependency(), and fixed it not adding
dependencies to parent groups. Renamed add() -> add_action() for clarity.
* Remove a couple of unused proto imports
* Migrate card info
* Migrate congrats, image occlusion, and tag editor
+ Fix imports for multi-word proto files.
* Migrate change-notetype
* Migrate deck options
* Bump target to es2020; simplify ts lib list
Have used caniuse.com to confirm Chromium 77, iOS 14.5 and the Chrome
on Android support the full es2017-es2020 features.
* Migrate import-csv
* Migrate i18n and fix missing output types in .js
* Migrate custom scheduling, and remove protobuf.js
To mostly maintain our old API contract, we make use of protobuf-es's
ability to convert to JSON, which follows the same format as protobuf.js
did. It doesn't cover all case: users who were previously changing the
variant of a type will need to update their code, as assigning to a new
variant no longer automatically removes the old one, which will cause an
error when we try to convert back from JSON. But I suspect the large majority
of users are adjusting the current variant rather than creating a new one,
and this saves us having to write proxy wrappers, so it seems like a
reasonable compromise.
One other change I made at the same time was to rename value->kind for
the oneofs in our custom study protos, as 'value' was easily confused
with the 'case/value' output that protobuf-es has.
With protobuf.js codegen removed, touching a proto file and invoking
./run drops from about 8s to 6s.
This closes#2043.
* Allow tree-shaking on protobuf types
* Display backend error messages in our ts alert()
* Make sourcemap generation opt-in for ts-run
Considerably slows down build, and not used most of the time.
This PR replaces the existing Python-driven sync server with a new one in Rust.
The new server supports both collection and media syncing, and is compatible
with both the new protocol mentioned below, and older clients. A setting has
been added to the preferences screen to point Anki to a local server, and a
similar setting is likely to come to AnkiMobile soon.
Documentation is available here: <https://docs.ankiweb.net/sync-server.html>
In addition to the new server and refactoring, this PR also makes changes to the
sync protocol. The existing sync protocol places payloads and metadata inside a
multipart POST body, which causes a few headaches:
- Legacy clients build the request in a non-deterministic order, meaning the
entire request needs to be scanned to extract the metadata.
- Reqwest's multipart API directly writes the multipart body, without exposing
the resulting stream to us, making it harder to track the progress of the
transfer. We've been relying on a patched version of reqwest for timeouts,
which is a pain to keep up to date.
To address these issues, the metadata is now sent in a HTTP header, with the
data payload sent directly in the body. Instead of the slower gzip, we now
use zstd. The old timeout handling code has been replaced with a new implementation
that wraps the request and response body streams to track progress, allowing us
to drop the git dependencies for reqwest, hyper-timeout and tokio-io-timeout.
The main other change to the protocol is that one-way syncs no longer need to
downgrade the collection to schema 11 prior to sending.
* Run cargo +nightly fmt
* Latest prost-build includes clippy workaround
* Tweak Rust protobuf imports
- Avoid use of stringify!(), as JetBrains editors get confused by it
- Stop merging all protobuf symbols into a single namespace
* Remove some unnecessary qualifications
Found via IntelliJ lint
* Migrate some asserts to assert_eq/ne
* Remove mention of node_modules exclusion
This no longer seems to be necessary after migrating away from Bazel,
and excluding it means TS/Svelte files can't be edited properly.
* Facilitate updating of hooks
- Add instructions in contributing.md
- Change addon_config_editor_will_update_json hook to work with the new
hookslib code
* Fix typo in docs
* Always run replaced hook
* Use lowercase list for typing
* Forbid defining both a replaced and a legacy hook
(for upgrading users, please see the notes at the bottom)
Bazel brought a lot of nice things to the table, such as rebuilds based on
content changes instead of modification times, caching of build products,
detection of incorrect build rules via a sandbox, and so on. Rewriting the build
in Bazel was also an opportunity to improve on the Makefile-based build we had
prior, which was pretty poor: most dependencies were external or not pinned, and
the build graph was poorly defined and mostly serialized. It was not uncommon
for fresh checkouts to fail due to floating dependencies, or for things to break
when trying to switch to an older commit.
For day-to-day development, I think Bazel served us reasonably well - we could
generally switch between branches while being confident that builds would be
correct and reasonably fast, and not require full rebuilds (except on Windows,
where the lack of a sandbox and the TS rules would cause build breakages when TS
files were renamed/removed).
Bazel achieves that reliability by defining rules for each programming language
that define how source files should be turned into outputs. For the rules to
work with Bazel's sandboxing approach, they often have to reimplement or
partially bypass the standard tools that each programming language provides. The
Rust rules call Rust's compiler directly for example, instead of using Cargo,
and the Python rules extract each PyPi package into a separate folder that gets
added to sys.path.
These separate language rules allow proper declaration of inputs and outputs,
and offer some advantages such as caching of build products and fine-grained
dependency installation. But they also bring some downsides:
- The rules don't always support use-cases/platforms that the standard language
tools do, meaning they need to be patched to be used. I've had to contribute a
number of patches to the Rust, Python and JS rules to unblock various issues.
- The dependencies we use with each language sometimes make assumptions that do
not hold in Bazel, meaning they either need to be pinned or patched, or the
language rules need to be adjusted to accommodate them.
I was hopeful that after the initial setup work, things would be relatively
smooth-sailing. Unfortunately, that has not proved to be the case. Things
frequently broke when dependencies or the language rules were updated, and I
began to get frustrated at the amount of Anki development time I was instead
spending on build system upkeep. It's now about 2 years since switching to
Bazel, and I think it's time to cut losses, and switch to something else that's
a better fit.
The new build system is based on a small build tool called Ninja, and some
custom Rust code in build/. This means that to build Anki, Bazel is no longer
required, but Ninja and Rust need to be installed on your system. Python and
Node toolchains are automatically downloaded like in Bazel.
This new build system should result in faster builds in some cases:
- Because we're using cargo to build now, Rust builds are able to take advantage
of pipelining and incremental debug builds, which we didn't have with Bazel.
It's also easier to override the default linker on Linux/macOS, which can
further improve speeds.
- External Rust crates are now built with opt=1, which improves performance
of debug builds.
- Esbuild is now used to transpile TypeScript, instead of invoking the TypeScript
compiler. This results in faster builds, by deferring typechecking to test/check
time, and by allowing more work to happen in parallel.
As an example of the differences, when testing with the mold linker on Linux,
adding a new message to tags.proto (which triggers a recompile of the bulk of
the Rust and TypeScript code) results in a compile that goes from about 22s on
Bazel to about 7s in the new system. With the standard linker, it's about 9s.
Some other changes of note:
- Our Rust workspace now uses cargo-hakari to ensure all packages agree on
available features, preventing unnecessary rebuilds.
- pylib/anki is now a PEP420 implicit namespace, avoiding the need to merge
source files and generated files into a single folder for running. By telling
VSCode about the extra search path, code completion now works with generated
files without needing to symlink them into the source folder.
- qt/aqt can't use PEP420 as it's difficult to get rid of aqt/__init__.py.
Instead, the generated files are now placed in a separate _aqt package that's
added to the path.
- ts/lib is now exposed as @tslib, so the source code and generated code can be
provided under the same namespace without a merging step.
- MyPy and PyLint are now invoked once for the entire codebase.
- dprint will be used to format TypeScript/json files in the future instead of
the slower prettier (currently turned off to avoid causing conflicts). It can
automatically defer to prettier when formatting Svelte files.
- svelte-check is now used for typechecking our Svelte code, which revealed a
few typing issues that went undetected with the old system.
- The Jest unit tests now work on Windows as well.
If you're upgrading from Bazel, updated usage instructions are in docs/development.md and docs/build.md. A summary of the changes:
- please remove node_modules and .bazel
- install rustup (https://rustup.rs/)
- install rsync if not already installed (on windows, use pacman - see docs/windows.md)
- install Ninja (unzip from https://github.com/ninja-build/ninja/releases/tag/v1.11.1 and
place on your path, or from your distro/homebrew if it's 1.10+)
- update .vscode/settings.json from .vscode.dist
