* Migrate build system to uv
Closes#3787, and is a step towards #3081 and #4022
This change breaks our PyOxidizer bundling process. While we probably
could update it to work with the new venvs & lockfile, my intention
is to use this as a base to try out a uv-based packager/installer.
Some notes about the changes:
- Use uv for python download + venv installation
- Drop python/requirements* in favour of pyproject files / uv.lock
- Bumped to latest Python 3.9 version. The move to 3.13 should be
a fairly trivial change when we're ready.
- Dropped the old write_wheel.py in favour of uv/hatchling. This has
the unfortunate side-effect of dropping leading zeros in our wheels,
which we could try hack around in the future.
- Switch to Qt 6.7 for the dev repo, as it's the first PyQt version
with a Linux/ARM WebEngine wheel.
- Unified our macOS deployment target with minimum required for ARM.
- Dropped unused fluent python files
- Dropped unused python license generation
- Dropped helpers to run under Qt 5, as our wheels were already
requiring Qt 6 to install.
* Build action to create universal uv binary
* Drop some PyOxidizer-related files
* Use Windows ARM64 cargo/node binaries during build
We can't provide ARM64 wheels to users yet due to #4079, but we can
at least speed up the build.
The rustls -> native-tls change on Windows is because ring requires
clang to compile for ARM64, and I figured it's best to keep our Windows
deps consistent. We already built the wheels with native-tls.
* Make libankihelper a universal library
We were shipping a single arch library in a purelib, leading to
breakages when running on a different platform.
* Use Python wheel for mpv/lame on Windows/Mac
This is convenient, but suboptimal on a Mac at the moment. The first
run of mpv will take a number of seconds for security checks to run,
and our mpv code ends up timing out, repeating the process each time.
Our installer stub will need to invoke mpv once first to get it validated.
We could address this by distributing the audio with the installer/stub,
or perhaps by putting the binaries in a .pkg file that's notarized+stapled
and then included in the wheel.
* Add some helper scripts to build a fully-locked wheel
* Initial macOS launcher prototype
* Add a hidden env var to preload our libs and audio helpers on macOS
* qt/bundle -> qt/launcher
- remove more of the old bundling code
- handle app icon
* Fat binary, notarization & dmg
* Publish wheels on testpypi for testing
* Use our Python pin for the launcher too
* Python cleanups
* Extend launcher to other platforms + more
- Switch to Qt 6.8 for repo default, as 6.7 depends on an older
libwebp/tiff which is unavailable on newer installs
- Drop tools/mac-x86, as we no longer need to test against Qt 5
- Add flags to cross compile wheels on Mac and Linux
- Bump glibc target to 2_36, building on Debian Stable
- Increase mpv timeout on macOS to allow for initial gatekeeper checks
- Ship both arm64 and amd64 uv on Linux, with a bash stub to pick
the appropriate arch.
* Fix pylint on Linux
* Fix failure to run from /usr/local/bin
* Remove remaining pyoxidizer refs, and clean up duplicate release folder
* Rust dep updates
- Rust 1.87 for now (1.88 due out in around a week)
- Nom looks involved, so I left it for now
- prost-reflect depends on a new prost version that got yanked
* Python 3.13 + dep updates
Updated protoc binaries + add helper in order to try fix build breakage.
Ended up being due to an AI-generated update to pip-system-certs that
was not reviewed carefully enough:
https://gitlab.com/alelec/pip-system-certs/-/issues/36
The updated mypy/black needed some tweaks to our files.
* Windows compilation fixes
* Automatically run Anki after installing on Windows
* Touch pyproject.toml upon install, so we check for updates
* Update Python deps
- urllib3 for CVE
- pip-system-certs got fixed
- markdown/pytest also updated
This reverts commit 04228de666.
Anki 25.01 Beta 1 revealed a bunch of regressions with the latest
Qt, and zero reports of improvements from it, so we'll be better off
holding off on it for now, and perhaps reporting the deadkeys issue
to Qt once we've got a proper reproduction process.
* Qt 6.8.1
Bumps minimum glibc to 2.35, and minimum macOS to 12
* Drop generation of Qt5 packaged build
Closes#3615
* Include qt6 requirements in aqt wheel; drop extra deps
* Fix aqt wheels growing over time
* fix: except only non-system-exiting exceptions
see https://youtu.be/zrVfY9SuO64
* chore: add myself to CONTRIBUTORS file
* refactor: explicitly specify possible exceptions
If an exception is not an Exception, there are only three options left.
see https://docs.python.org/3/library/exceptions.html#exception-hierarchy
* refactor: use BaseException for fallback
Co-authored-by: Damien Elmes <dae@users.noreply.github.com>
* chore: add myself to contributors
* chore: add myself to CONTRIBUTORS file
* refactor: use newer type hints for Union/Optional
* refactor: fix deprecated type annotations
use collections.abc rather than typing
* refactor: use lower letter type annotations
* style: reformat with black
* refactor: remove unused imports
* refactor: add missing imports for type hints
* fixup! refactor: use newer type hints for Union/Optional
* fix: add missing imports for type annotations
* fixup! refactor: use newer type hints for Union/Optional
* fixup! style: reformat with black
* refactor: fix remaining imports re: type hints
If you have an add-on that hasn't been updated yet, please contact
the add-on author. You can temporarily restore functionality by defining
an ENABLE_QT5_COMPAT=1 env var.
* Expose video driver options for Qt6
* Default to d3d11/metal/vulkan on Qt6
* Remove `pass`
* Add missing word to existing translation
* Default to OpenGL on Linux
* Exclude Vulkan from macOS
* Label default drivers
(for upgrading users, please see the notes at the bottom)
Bazel brought a lot of nice things to the table, such as rebuilds based on
content changes instead of modification times, caching of build products,
detection of incorrect build rules via a sandbox, and so on. Rewriting the build
in Bazel was also an opportunity to improve on the Makefile-based build we had
prior, which was pretty poor: most dependencies were external or not pinned, and
the build graph was poorly defined and mostly serialized. It was not uncommon
for fresh checkouts to fail due to floating dependencies, or for things to break
when trying to switch to an older commit.
For day-to-day development, I think Bazel served us reasonably well - we could
generally switch between branches while being confident that builds would be
correct and reasonably fast, and not require full rebuilds (except on Windows,
where the lack of a sandbox and the TS rules would cause build breakages when TS
files were renamed/removed).
Bazel achieves that reliability by defining rules for each programming language
that define how source files should be turned into outputs. For the rules to
work with Bazel's sandboxing approach, they often have to reimplement or
partially bypass the standard tools that each programming language provides. The
Rust rules call Rust's compiler directly for example, instead of using Cargo,
and the Python rules extract each PyPi package into a separate folder that gets
added to sys.path.
These separate language rules allow proper declaration of inputs and outputs,
and offer some advantages such as caching of build products and fine-grained
dependency installation. But they also bring some downsides:
- The rules don't always support use-cases/platforms that the standard language
tools do, meaning they need to be patched to be used. I've had to contribute a
number of patches to the Rust, Python and JS rules to unblock various issues.
- The dependencies we use with each language sometimes make assumptions that do
not hold in Bazel, meaning they either need to be pinned or patched, or the
language rules need to be adjusted to accommodate them.
I was hopeful that after the initial setup work, things would be relatively
smooth-sailing. Unfortunately, that has not proved to be the case. Things
frequently broke when dependencies or the language rules were updated, and I
began to get frustrated at the amount of Anki development time I was instead
spending on build system upkeep. It's now about 2 years since switching to
Bazel, and I think it's time to cut losses, and switch to something else that's
a better fit.
The new build system is based on a small build tool called Ninja, and some
custom Rust code in build/. This means that to build Anki, Bazel is no longer
required, but Ninja and Rust need to be installed on your system. Python and
Node toolchains are automatically downloaded like in Bazel.
This new build system should result in faster builds in some cases:
- Because we're using cargo to build now, Rust builds are able to take advantage
of pipelining and incremental debug builds, which we didn't have with Bazel.
It's also easier to override the default linker on Linux/macOS, which can
further improve speeds.
- External Rust crates are now built with opt=1, which improves performance
of debug builds.
- Esbuild is now used to transpile TypeScript, instead of invoking the TypeScript
compiler. This results in faster builds, by deferring typechecking to test/check
time, and by allowing more work to happen in parallel.
As an example of the differences, when testing with the mold linker on Linux,
adding a new message to tags.proto (which triggers a recompile of the bulk of
the Rust and TypeScript code) results in a compile that goes from about 22s on
Bazel to about 7s in the new system. With the standard linker, it's about 9s.
Some other changes of note:
- Our Rust workspace now uses cargo-hakari to ensure all packages agree on
available features, preventing unnecessary rebuilds.
- pylib/anki is now a PEP420 implicit namespace, avoiding the need to merge
source files and generated files into a single folder for running. By telling
VSCode about the extra search path, code completion now works with generated
files without needing to symlink them into the source folder.
- qt/aqt can't use PEP420 as it's difficult to get rid of aqt/__init__.py.
Instead, the generated files are now placed in a separate _aqt package that's
added to the path.
- ts/lib is now exposed as @tslib, so the source code and generated code can be
provided under the same namespace without a merging step.
- MyPy and PyLint are now invoked once for the entire codebase.
- dprint will be used to format TypeScript/json files in the future instead of
the slower prettier (currently turned off to avoid causing conflicts). It can
automatically defer to prettier when formatting Svelte files.
- svelte-check is now used for typechecking our Svelte code, which revealed a
few typing issues that went undetected with the old system.
- The Jest unit tests now work on Windows as well.
If you're upgrading from Bazel, updated usage instructions are in docs/development.md and docs/build.md. A summary of the changes:
- please remove node_modules and .bazel
- install rustup (https://rustup.rs/)
- install rsync if not already installed (on windows, use pacman - see docs/windows.md)
- install Ninja (unzip from https://github.com/ninja-build/ninja/releases/tag/v1.11.1 and
place on your path, or from your distro/homebrew if it's 1.10+)
- update .vscode/settings.json from .vscode.dist
* Use horizontal orientation on browser splitter by default
* Add View menu action to toggle browser orientation
* Add shortcut for toggleOrientation action
based on the most popular add-on.
* Try to fix typing issue
* Make orientation respond to aspect ratio
aspect ratio < 1 means vertical orientation, >= 1 horizontal
* Implement three-way switch for browser orientation
* Fix typing
* Add separator before QWidgetAction
* Use submenu instead of widget and adjust enum
* Add accelerators; move non-accelerator strings into separate .ftl (dae)
* Move BrowserLayout to its own file (dae)
* Alias PyQt5 to PyQt6 on PyQt6 builds
Restores basic compatibility with PyQt5 add-ons
* Register QtCore early to work around sip error
* Monkey-patch unscoped enums that are in use by add-ons back in
Enums whose namespace moved with PyQt6 were determined using the tooling in https://github.com/qutebrowser/qutebrowser/issues/5904
Relevant enums for the Anki add-on ecosystem were found by grepping through all AnkiWeb add-ons and a selection of GitHub-released add-ons.
* Add full Qt.Key namespace
Maintains compatibility with add-ons that allow specifying key bindings via Qt.Key enums
* Reintroduce PyQt6.Qt as an alias for QtCore.Qt
* Alias classes shifted from QtWidgets to QtGui
* Add missing enums
Adds ≈200 enums that were missed during the initial grep
* Map exec_ calls to exec
* Tweak section headers
* Fix QtWebEngineWidgets imports failing due to delayed import
Addesses: "QtWebEngineWidgets must be imported before a QCoreApplication instance is created"
* Register additional aliases for top-level Qt modules
Given how we have had to deal with side-effects when not registering other aliased imports ahead of time, it seems safer to also register the remaining few with sys.modules.
* Handle calls to deprecated PyQt resource API graciously
* Create QtWebEngineWidgets aliases for classes moved to QtWebEngineCore
* Alias QShortcut
* Restore QWebEnginePage.view()
* Alias sip to PyQt6.sip
* Alias QtCore.QRegExp to QtCore.QRegularExpression
* Restructure aqt.qt into package
Pre-requirement for aliasing the PyQt5.Qt namespace correctly.
Should hopefully also make it easier to keep an overview as Qt-compat-related modules were proliferating.
* Properly alias PyQt5.Qt
PyQt5.Qt used to serve as a common namespace for all Qt classes, not just QtCore.Qt.*
While this changes does not make all classes accessible via PyQt5.Qt, it does so for the most important Qt submodules, which should cover most add-on breakages.
* Simplify Qt resource system legacy handling
* Also alias PyQt6.Qt
Covers imports of the form `from PyQt5 import import Qt` (due to previous aliasing of PyQt5 to PyQt6)
* Add missing enums
Better approach to grepping through add-ons yielded additional hits
* Run formatters
* Satisfy pylint
