* Migrate build system to uv
Closes#3787, and is a step towards #3081 and #4022
This change breaks our PyOxidizer bundling process. While we probably
could update it to work with the new venvs & lockfile, my intention
is to use this as a base to try out a uv-based packager/installer.
Some notes about the changes:
- Use uv for python download + venv installation
- Drop python/requirements* in favour of pyproject files / uv.lock
- Bumped to latest Python 3.9 version. The move to 3.13 should be
a fairly trivial change when we're ready.
- Dropped the old write_wheel.py in favour of uv/hatchling. This has
the unfortunate side-effect of dropping leading zeros in our wheels,
which we could try hack around in the future.
- Switch to Qt 6.7 for the dev repo, as it's the first PyQt version
with a Linux/ARM WebEngine wheel.
- Unified our macOS deployment target with minimum required for ARM.
- Dropped unused fluent python files
- Dropped unused python license generation
- Dropped helpers to run under Qt 5, as our wheels were already
requiring Qt 6 to install.
* Build action to create universal uv binary
* Drop some PyOxidizer-related files
* Use Windows ARM64 cargo/node binaries during build
We can't provide ARM64 wheels to users yet due to #4079, but we can
at least speed up the build.
The rustls -> native-tls change on Windows is because ring requires
clang to compile for ARM64, and I figured it's best to keep our Windows
deps consistent. We already built the wheels with native-tls.
* Make libankihelper a universal library
We were shipping a single arch library in a purelib, leading to
breakages when running on a different platform.
* Use Python wheel for mpv/lame on Windows/Mac
This is convenient, but suboptimal on a Mac at the moment. The first
run of mpv will take a number of seconds for security checks to run,
and our mpv code ends up timing out, repeating the process each time.
Our installer stub will need to invoke mpv once first to get it validated.
We could address this by distributing the audio with the installer/stub,
or perhaps by putting the binaries in a .pkg file that's notarized+stapled
and then included in the wheel.
* Add some helper scripts to build a fully-locked wheel
* Initial macOS launcher prototype
* Add a hidden env var to preload our libs and audio helpers on macOS
* qt/bundle -> qt/launcher
- remove more of the old bundling code
- handle app icon
* Fat binary, notarization & dmg
* Publish wheels on testpypi for testing
* Use our Python pin for the launcher too
* Python cleanups
* Extend launcher to other platforms + more
- Switch to Qt 6.8 for repo default, as 6.7 depends on an older
libwebp/tiff which is unavailable on newer installs
- Drop tools/mac-x86, as we no longer need to test against Qt 5
- Add flags to cross compile wheels on Mac and Linux
- Bump glibc target to 2_36, building on Debian Stable
- Increase mpv timeout on macOS to allow for initial gatekeeper checks
- Ship both arm64 and amd64 uv on Linux, with a bash stub to pick
the appropriate arch.
* Fix pylint on Linux
* Fix failure to run from /usr/local/bin
* Remove remaining pyoxidizer refs, and clean up duplicate release folder
* Rust dep updates
- Rust 1.87 for now (1.88 due out in around a week)
- Nom looks involved, so I left it for now
- prost-reflect depends on a new prost version that got yanked
* Python 3.13 + dep updates
Updated protoc binaries + add helper in order to try fix build breakage.
Ended up being due to an AI-generated update to pip-system-certs that
was not reviewed carefully enough:
https://gitlab.com/alelec/pip-system-certs/-/issues/36
The updated mypy/black needed some tweaks to our files.
* Windows compilation fixes
* Automatically run Anki after installing on Windows
* Touch pyproject.toml upon install, so we check for updates
* Update Python deps
- urllib3 for CVE
- pip-system-certs got fixed
- markdown/pytest also updated
This commit explains how to calls a method implemented in a language
from a different language.
This explains how to declare the RPCs, how to call them and how to
implement them. This is based on examples of code at main at the time
of writting. I used permalink to ensure that the links remains
relevant even if the specific examples change later.
The last section is about the special case of calling TypeScript from
Python, which does not use RPC but is still relevant in a bridge
document.
This commit also add a paragraph explaining what protobuf is in the
protobuf documentation, so that new contributors who don't know what
protobuf is can understand why we use it.
* pass --locked to cargo invocation
* update Dockerfile.distroless as well
Co-authored-by: Simon <8466614+SimonBaars@users.noreply.github.com>
---------
Co-authored-by: Simon <8466614+SimonBaars@users.noreply.github.com>
Hardcode them to:
SYNC_PORT=8080
SYNC_BASE=/anki_data
If these env variables are passed into the container with different values,
they are ignored.
The reasons is if the user modifies SYNC_BASE they risk data loss since
anki-sync-server will no longer write data into the volume. If they change
SYNC_PORT they need to also change it when mapping this internal port to the
external port of the container, which could be confusing plus it has no benefit
to allow this since it's always possible to change the external port even if
the internal port is fixed to 8080 (e.g. `-p 1234:8080`).
In both cases there is no benefit to making these values configurable and there
are risks associated.
Unfortunately there is no easy way of implementing this for the
Dockerfile.distroless so it's up to the user not to modify these values.
PUID and PGID are optional env variables to specify the user and group id of
the user that the anki-sync-server process should run with.
This gives more flexibility for solving permission problems with volumes and is
a common pattern for Docker images (e.g. see here:
https://docs.linuxserver.io/general/understanding-puid-and-pgid/)
The anki-sync-server process will write any files with the permissions of the
user it's running with, which can be a problem when you need to access those
files from outside the container or when they are being written into a bind
mount that is owned by a particular user on the host system.
To be able to implement this the entrypoint.sh needs to run as root (since it
needs to create a user and change file permissions). anki-sync-server then
needs to be started with the user 'anki', which is why the new dependency
'su-exec' is required. The user 'anki' and group 'anki-group' can no longer be
created at image build time because then their ids would be fixed.
Also update the build instructions to require building the Docker image inside
the directory where the Dockerfile resides since the build now needs to copy
the entrypoint.sh and it seems wrong the specify the path
docs/syncserver/entrypoint.sh inside the Dockerfile.
Now that an ARM wheel is on PyPI, we no longer need to rely on a
system PyQt to build on ARM. The install is skipped when PYTHONPATH
is set, so older distros with glibc <2.39 can continue to use the
system packages instead.
Otherwise data would be lost by default when removing (or re-creating) a
container.
It would be possible to expose the default directory (e.g.
/home/anki/.syncserver) but it would be different for the two Dockerfiles and
less convenient for users of the Docker container to specify such a long path
when naming their volumes.
Setting the permissions is necessary since anki will be running with 'anki'
user permissions inside the container.
* Qt 6.8.1
Bumps minimum glibc to 2.35, and minimum macOS to 12
* Drop generation of Qt5 packaged build
Closes#3615
* Include qt6 requirements in aqt wheel; drop extra deps
* Fix aqt wheels growing over time
* Add myself to CONTRIBUTORS file
* replace localhost with 127.0.0.1 in syncserver Dockerfile
The healthcheck was failing, presumably because localhost was resolving to ::1
(IPv6), as detailed in this issue: https://github.com/maildev/maildev/pull/500
* docs(docker): Change suggested version numbre
* deps(docker): Bump rust to 1.83.0 and alpine to 3.21.0
* deps(docker): Bump rust to 1.83.0
* CONTRIBUTORS: Add my name
* Add myself to CONTRIBUTORS file
* avoid warning by setting SYNC_PORT as ARG in Dockerfile
1 warning found (use docker --debug to expand):
- UndefinedVar: Usage of undefined variable '$SYNC_PORT'
- rslib(http_server): add `is_running()` method
- rslib(sync): introduce `--healthcheck` argument for health probe in distroless
- doc(syncserver): add table comparing Dockerfile and Dockerfile.distroless
- Expand cross-platform support with distroless
- add `Dockerfile.distroless`
- Dockerfile: bump rust `1.79` to `1.80.1`
- Dockerfile: bump alpine `3.20` to `3.20.2`
Note: Implemented an internal health check because distroless images do not include curl, which is used to reduce image size and attack surface. For more details, see https://blog.sixeyed.com/docker-healthchecks-why-not-to-use-curl-or-iwr/https://github.com/GoogleContainerTools/distroless
fix: failed: check:format:rust
typo
remove extra space
fix failed:check:format:rust
update doc
fetch `host` and `port` using envy
fix: failed: check:format:rust
Update doc + add dockerignore
- dockerignore: This helps avoid sending unwanted files and directories to the builder
- add new line
- I am still experimenting cross platform compilation, I am getting
4.337 From https://github.com/ankitects/rust-url
4.337 * [new ref] bb930b8d089f4d30d7d19c12e54e66191de47b88 -> refs/commit/bb930b8d089f4d30d7d19c12e54e66191de47b88
4.397 error: failed to get `percent-encoding-iri` as a dependency of package `anki v0.0.0 (/app/rslib)`
still checking what could be the issue
fix: failed: check:format:dprint
* Update base images and introduce health endpoint
sync-server: introduce `/health` endpoint to check if the service is reachable.
bump(alpine): bump alpine base image from `3.19` to `3.20`
bump(rust): bump rust-alpine build image from `1.76` to `1.79`
* fix cargo fmt
* add allow clippy::extra_unused_type_parameters
* Remove unused type param (dae)
* Route /health directly (dae)
* Fix for latest axum (dae)
* Simplify the offline build
The two environment variables OFFLINE_BUILD and NO_VENV jointly provide
the ability to build Anki fully offline. This commit boils them down
into just one, namely OFFLINE_BUILD.
The rationale being that first, OFFLINE_BUILD implies the use of
a custom non-networked Python environment.
Second, building Anki with a custom Python environment in a networked
setting is a use case, that we currently do not support.
Developers in need of such a solution may want to give containerized
development environments a try. Users could also look into building
Anki fully offline instead.
* Add documentation for offline builds.
* Add support for offline generation of Sphinx documentation.
Control installation of Sphinx dependencies via the network through the
OFFLINE_BUILD environment variable.
* Add documentation for offline generation of Sphinx documentation.
* Add `extra` directory as a designated ignored folder
Excludes `extra/` from version tracking, file formatters, and file checks.
* Remove pytest cache from exclusion rules
Python test discovery is easy enough to disable for the workspace in VS Code's settings and pytest does not serve any purpose in the context of the project anyway.
* Fix .no-reduce-motion missing from graphs spinner, and not being honored
* Begin migration from protobuf.js -> protobuf-es
Motivation:
- Protobuf-es has a nicer API: messages are represented as classes, and
fields which should exist are not marked as nullable.
- As it uses modules, only the proto messages we actually use get included
in our bundle output. Protobuf.js put everything in a namespace, which
prevented tree-shaking, and made it awkward to access inner messages.
- ./run after touching a proto file drops from about 8s to 6s on my machine. The tradeoff
is slower decoding/encoding (#2043), but that was mainly a concern for the
graphs page, and was unblocked by
37151213cd
Approach/notes:
- We generate the new protobuf-es interface in addition to existing
protobuf.js interface, so we can migrate a module at a time, starting
with the graphs module.
- rslib:proto now generates RPC methods for TS in addition to the Python
interface. The input-arg-unrolling behaviour of the Python generation is
not required here, as we declare the input arg as a PlainMessage<T>, which
marks it as requiring all fields to be provided.
- i64 is represented as bigint in protobuf-es. We were using a patch to
protobuf.js to get it to output Javascript numbers instead of long.js
types, but now that our supported browser versions support bigint, it's
probably worth biting the bullet and migrating to bigint use. Our IDs
fit comfortably within MAX_SAFE_INTEGER, but that may not hold for future
fields we add.
- Oneofs are handled differently in protobuf-es, and are going to need
some refactoring.
Other notable changes:
- Added a --mkdir arg to our build runner, so we can create a dir easily
during the build on Windows.
- Simplified the preference handling code, by wrapping the preferences
in an outer store, instead of a separate store for each individual
preference. This means a change to one preference will trigger a redraw
of all components that depend on the preference store, but the redrawing
is cheap after moving the data processing to Rust, and it makes the code
easier to follow.
- Drop async(Reactive).ts in favour of more explicit handling with await
blocks/updating.
- Renamed add_inputs_to_group() -> add_dependency(), and fixed it not adding
dependencies to parent groups. Renamed add() -> add_action() for clarity.
* Remove a couple of unused proto imports
* Migrate card info
* Migrate congrats, image occlusion, and tag editor
+ Fix imports for multi-word proto files.
* Migrate change-notetype
* Migrate deck options
* Bump target to es2020; simplify ts lib list
Have used caniuse.com to confirm Chromium 77, iOS 14.5 and the Chrome
on Android support the full es2017-es2020 features.
* Migrate import-csv
* Migrate i18n and fix missing output types in .js
* Migrate custom scheduling, and remove protobuf.js
To mostly maintain our old API contract, we make use of protobuf-es's
ability to convert to JSON, which follows the same format as protobuf.js
did. It doesn't cover all case: users who were previously changing the
variant of a type will need to update their code, as assigning to a new
variant no longer automatically removes the old one, which will cause an
error when we try to convert back from JSON. But I suspect the large majority
of users are adjusting the current variant rather than creating a new one,
and this saves us having to write proxy wrappers, so it seems like a
reasonable compromise.
One other change I made at the same time was to rename value->kind for
the oneofs in our custom study protos, as 'value' was easily confused
with the 'case/value' output that protobuf-es has.
With protobuf.js codegen removed, touching a proto file and invoking
./run drops from about 8s to 6s.
This closes#2043.
* Allow tree-shaking on protobuf types
* Display backend error messages in our ts alert()
* Make sourcemap generation opt-in for ts-run
Considerably slows down build, and not used most of the time.
This PR replaces the existing Python-driven sync server with a new one in Rust.
The new server supports both collection and media syncing, and is compatible
with both the new protocol mentioned below, and older clients. A setting has
been added to the preferences screen to point Anki to a local server, and a
similar setting is likely to come to AnkiMobile soon.
Documentation is available here: <https://docs.ankiweb.net/sync-server.html>
In addition to the new server and refactoring, this PR also makes changes to the
sync protocol. The existing sync protocol places payloads and metadata inside a
multipart POST body, which causes a few headaches:
- Legacy clients build the request in a non-deterministic order, meaning the
entire request needs to be scanned to extract the metadata.
- Reqwest's multipart API directly writes the multipart body, without exposing
the resulting stream to us, making it harder to track the progress of the
transfer. We've been relying on a patched version of reqwest for timeouts,
which is a pain to keep up to date.
To address these issues, the metadata is now sent in a HTTP header, with the
data payload sent directly in the body. Instead of the slower gzip, we now
use zstd. The old timeout handling code has been replaced with a new implementation
that wraps the request and response body streams to track progress, allowing us
to drop the git dependencies for reqwest, hyper-timeout and tokio-io-timeout.
The main other change to the protocol is that one-way syncs no longer need to
downgrade the collection to schema 11 prior to sending.
* Run cargo +nightly fmt
* Latest prost-build includes clippy workaround
* Tweak Rust protobuf imports
- Avoid use of stringify!(), as JetBrains editors get confused by it
- Stop merging all protobuf symbols into a single namespace
* Remove some unnecessary qualifications
Found via IntelliJ lint
* Migrate some asserts to assert_eq/ne
* Remove mention of node_modules exclusion
This no longer seems to be necessary after migrating away from Bazel,
and excluding it means TS/Svelte files can't be edited properly.
