* Allow im-/exporting with or without deck configs
Closes#2777.
* Enable webengine remote debugging in launch.json
* Reset deck limits and counts based on scheduling
Also:
- Fix `deck.common` not being reset.
- Apply all logic only depending on the source collection in the
gathering stage.
- Skip checking for scheduling and only act based on whether the call
wants scheduling. Preservation of filtered decks also depends on all
original decks being included.
- Fix check_ids() not covering revlog.
* Fix importing legacy filtered decks w/o scheduling
* Disable 'include deck options' by default, and fix tab order (dae)
* deck options > deck presets (dae)
This reverts commit e1e0f2e1bd.
The previous change breaks the assumption on this line:
DeckKind::Filtered(_) => unreachable!(),
This will likely need a bigger refactor to handle this properly, and separate
out importing of deck configs from other scheduling data.
* Remove v1/v2 support from deck list
* Remove v1/v2 support from most routines and show error
* Remove scheduler_version from preferences
* Fix formatting
* Remove v1/v2 conditionals from Python code
* Fix legacy importer
* Remove legacy hooks
* Add missing scheduler checks
* Remove V2 logic from deck options screen
* Remove the review_did_undo hook
* Restore ability to open old options with shift (dae)
* Refactor import apkg tests
* Merge conflicting notetypes regardless of id match
Original ids are a new thing, and we need to handle previous remappings.
This is done separately from the conflict resolution for notetypes with
matching ids, because 1) we need to look at the notes to determine
conflicts, and 2) we don't want to change the notetype of *all* existing
notes with the conflicting notetype. The main reason is that for 2
existing notes with the same noteype, their incoming counterparts could
have *different* notetypes. So to get rid of all conflicts, they must be
resolved on a note-by-note basis.
* Delete merged, now unused notetypes
* Remember original id when importing notetype
* Reuse notetypes with matching original id
* Add field and template ids
* Enable merging imported notetypes
* Fix test
Note should be updated if the incoming note's notetype is
remapped to the existing note's notetype.
On the other hand, it should be skipped if its notetype id is mapped
to some new notetype.
* Change field and template ids to i32
* Add merge notetypes flag to proto message
* Add dialog for apkg import
* Move HelpModal into components
* Generalize import dialog
* Move SettingTitle into components
* Add help modal to ImportAnkiPackagePage
* Move SwitchRow into components
* Fix backend method import
* Make testable in browser
* Fix broken modal
* Wrap in container and fix margins
* Update commented Anki version of new proto fields
* Check ids when comparing notetype schemas
* Add tooltip for merging notetypes.
* Allow updating notes regardless of mtime
* Gitignore yarn-error.log
* Allow updating notetypes regardless of mtime
* Fix apkg help carousel
* Use i64s for template and field ids
* Add option to omit importing scheduling info
* Restore last settings in apkg import dialog
* Display error when getting metadata in webview
* Update manual links for apkg importing
* Apply suggestions from code review
Co-authored-by: Damien Elmes <dae@users.noreply.github.com>
* Omit schduling -> Import all cards as new cards
* Tweak importing-update-notes-help
* UpdateCondition → ImportAnkiPackageUpdateCondition
* Load keyboard.ftl
* Skip updating dupes in 'update alwyas' case
* Explain more when merging notetypes is required
* "omit scheduling" → "with scheduling"
* Skip updating notetype dupes if 'update always'
* Merge duplicated notetypes from previous imports
* Fix rebase aftermath
* Fix panic when merging
* Clarify 'update notetypes' help
* Mention 'merge notetypes' in the log
* Add a test which covers the previously panicking path
* Use nested ftl messages to ensure consistency
* Make order of merged fields deterministic
* Rewrite test to trigger panic
* Update version comment on new fields
* Implement import log screen in Svelte
* Show filename in import log screen title
* Remove unused NoteRow property
* Show number of imported notes
* Use a single nid expression
* Use 'count' as variable name for consistency
* Import from @tslib/backend instead
* Fix summary_template typing
* Fix clippy warning
* Apply suggestions from code review
* Fix imports
* Contents -> Fields
* Increase max length of browser search bar
https://github.com/ankitects/anki/pull/2568/files#r1255227035
* Fix race condition in Bootstrap tooltip destruction
https://github.com/twbs/bootstrap/issues/37474
* summary_template -> summaryTemplate
* Make show link a button
* Run import ops on Svelte side
* Fix geometry not being restored in CSV Import page
* Make VirtualTable fill available height
* Keep CSV dialog modal
* Reword importing-existing-notes-skipped
* Avoid mentioning matching based on first field
* Change tick and cross icons
* List skipped notes last
* Pure CSS spinner
* Move set_wants_abort() call to relevant dialogs
* Show number of imported cards
* Remove bold from first sentence and indent summaries
* Update UI after import operations
* Add close button to import log page
Also make virtual table react to resize event.
* Fix typing
* Make CSV dialog non-modal again
Otherwise user can't interact with browser window.
* Update window modality after import
* Commit DB and update undo actions after import op
* Split frontend proto into separate file, so backend can ignore it
Currently the automatically-generated frontend RPC methods get placed in
'backend.js' with all the backend methods; we could optionally split them
into a separate 'frontend.js' file in the future.
* Migrate import_done from a bridgecmd to a HTTP request
* Update plural form of importing-notes-added
* Move import response handling to mediasrv
* Move task callback to script section
* Avoid unnecessary :global()
* .log cannot be missing if result exists
* Move import log search handling to mediasrv
* Type common params of ImportLogDialog
* Use else if
* Remove console.log()
* Add way to test apkg imports in new log screen
* Remove unused import
* Get actual card count for CSV imports
* Use import type
* Fix typing error
* Ignore import log when checking for changes in Python layer
* Apply suggestions from code review
* Remove imported card count for now
* Avoid non-null assertion in assignment
* Change showInBrowser to take an array of notes
* Use dataclasses for import log args
* Simplify ResultWithChanges in TS
* Only abort import when window is modal
* Fix ResultWithChanges typing
* Fix Rust warnings
* Only log one duplicate per incoming note
* Update wording about note updates
* Remove caveat about found_notes
* Reduce font size
* Remove redundant map
* Give credit to loading.io
* Remove unused line
---------
Co-authored-by: RumovZ <gp5glkw78@relay.firefox.com>
Instead of flattening the output (which was missing FrontSide), alter
the behaviour of render() instead. The non-partial output is now exposed
via Protobuf, so the non-Python clients can take advantage of it.
* Automatically elide empty inputs and outputs to backend methods
* Refactor service generation
Despite the fact that the majority of our Protobuf service methods require
an open collection, they were not accessible with just a Collection
object. To access the methods (e.g. because we haven't gotten around to
exposing the correct API in Collection yet), you had to wrap the collection
in a Backend object, and pay a mutex-acquisition cost for each call, even
if you have exclusive access to the object.
This commit migrates the majority of service methods to the Collection, so
they can now be used directly, and improves the ergonomics a bit at the
same time.
The approach taken:
- The service generation now happens in rslib instead of anki_proto, which
avoids the need for trait constraints and associated types.
- Service methods are assumed to be collection-based by default. Instead of
implementing the service on Backend, we now implement it on Collection, which
means our methods no longer need to use self.with_col(...).
- We automatically generate methods in Backend which use self.with_col() to
delegate to the Collection method.
- For methods that are only appropriate for the backend, we add a flag in
the .proto file. The codegen uses this flag to write the method into a
BackendFooService instead of FooService, which the backend implements.
- The flag can also allows us to define separate implementations for collection
and backend, so we can e.g. skip the collection mutex in the i18n service
while also providing the service on a collection.
Previously it was Backend's responsibility to store the last progress,
and when calling routines in Collection, one had to construct and pass
in a Fn, which wasn't the most ergonomic. This PR adds the last progress
state to the collection, so that the routines no longer need a separate
progress arg, and makes some other tweaks to improve ergonomics.
ThrottlingProgressHandler has been tweaked so that it now stores the
current state, so that callers don't need to store it separately. When
a long-running routine starts, it calls col.new_progress_handler(),
which automatically initializes the data to defaults, and updates the
shared UI state, so we no longer need to manually update the state at
the start of an operation.
The backend shares the Arc<Mutex<>> with the collection, so it can get
at the current state, and so we can update the state when importing a
backup.
Other tweaks:
- The current Incrementor was awkward to use in the media check, which
uses a single incrementing value across multiple method calls, so I've
added a simpler alternative for such cases. The old incrementor method
has been kept, but implemented directly on ThrottlingProgressHandler.
- The full sync code was passing the progress handler in a complicated
way that may once have been required, but no longer is.
- On the Qt side, timers are now stopped before deletion, or they keep
running for a few seconds.
- I left the ChangeTracker using a closure, as it's used for both importing
and syncing.
Will be handy to use it in our other scripts in the future too - thanks
Rumo!
Results of benchmarking ./run before and after these crate splits:
- Touching a proto file leads to a slight increase: about +90ms
- Touching an rslib file leads to a bigger decrease, as there's less to
recompile: about -700ms
And ./ninja test is even better: about +200ms and -3800ms.
Due to the orphan rule, this meant removing our usages of impl ProtoStruct,
or converting them to a trait when they were used commonly.
rslib now directly references anki_proto and anki_i18n, instead of
'pub use'-ing them, and we can put the generated files back in OUT_DIR.
* Default to current deck in csv import if settings allow it
Reuses defaults_for_adding(). In the future we might also want to update
the last deck/notetype on successful completion, if entries weren't
specified in the file.
https://forums.ankiweb.net/t/importing-new-notes-to-wrong-deck-in-anki-2-1-63/30598
* Address review feedback from Rumo
- Ensure our unit tests cover the non-legacy path, and check file
contents and not just existence.
- Pass `compressed` into copy_and_ensure_sha1_set(). We can't just
test for self.sha1.is_none(), as that fails in the case where an existing
media file was found in the legacy path.
https://forums.ankiweb.net/t/anki-media-file-corruption-when-exporting-png-files/30315
* Fix file extension not being appended on export
Regressed in #2427
* Improve import messaging when notetype has changed
- If the local notes are up to date, we don't need to warn about the
changed notetype, as no updates are required.
- Make it clearer that a changed notetype only affects updates.
Will update the docs as well.
* Move open_test_collection into Collection test impl
* Fix invalid ids when checking database
* Report fixed invalid ids
* Improve message when trying to export invalid ids
Also move ImportError due to namespace conflicts with snafu macro.
* Take a human name in DeckAdder::new
* Mention timestamps in the db check message (dae)
Will help to correlate the fix with the message shown when importing/
exporting.
* Add CardAdder test helper
* Add option to have new cards ignore the review limit
Also entails a lot of refactoring because the old code was deeply
coupled to the previous behaviour.
* Add global option to ignore review limit
* Refactor decrementation
* Unify testing
This PR replaces the existing Python-driven sync server with a new one in Rust.
The new server supports both collection and media syncing, and is compatible
with both the new protocol mentioned below, and older clients. A setting has
been added to the preferences screen to point Anki to a local server, and a
similar setting is likely to come to AnkiMobile soon.
Documentation is available here: <https://docs.ankiweb.net/sync-server.html>
In addition to the new server and refactoring, this PR also makes changes to the
sync protocol. The existing sync protocol places payloads and metadata inside a
multipart POST body, which causes a few headaches:
- Legacy clients build the request in a non-deterministic order, meaning the
entire request needs to be scanned to extract the metadata.
- Reqwest's multipart API directly writes the multipart body, without exposing
the resulting stream to us, making it harder to track the progress of the
transfer. We've been relying on a patched version of reqwest for timeouts,
which is a pain to keep up to date.
To address these issues, the metadata is now sent in a HTTP header, with the
data payload sent directly in the body. Instead of the slower gzip, we now
use zstd. The old timeout handling code has been replaced with a new implementation
that wraps the request and response body streams to track progress, allowing us
to drop the git dependencies for reqwest, hyper-timeout and tokio-io-timeout.
The main other change to the protocol is that one-way syncs no longer need to
downgrade the collection to schema 11 prior to sending.
The Rust community appear to have converged on tracing - it's used by
the Rust compiler, and receives close to 10x the number of downloads
that slog does. Its API is more ergonomic, and it does a much nicer
job with async rust.
To make this change, we no longer pass around explicit loggers, and rely
on a globally-registered one. The log file location has been changed
from one in each profile folder to a single one in the base folder. This
will remain empty for most users, since only errors are logged by default,
but may be useful for debugging future changes.
* Run cargo +nightly fmt
* Latest prost-build includes clippy workaround
* Tweak Rust protobuf imports
- Avoid use of stringify!(), as JetBrains editors get confused by it
- Stop merging all protobuf symbols into a single namespace
* Remove some unnecessary qualifications
Found via IntelliJ lint
* Migrate some asserts to assert_eq/ne
* Remove mention of node_modules exclusion
This no longer seems to be necessary after migrating away from Bazel,
and excluding it means TS/Svelte files can't be edited properly.
(for upgrading users, please see the notes at the bottom)
Bazel brought a lot of nice things to the table, such as rebuilds based on
content changes instead of modification times, caching of build products,
detection of incorrect build rules via a sandbox, and so on. Rewriting the build
in Bazel was also an opportunity to improve on the Makefile-based build we had
prior, which was pretty poor: most dependencies were external or not pinned, and
the build graph was poorly defined and mostly serialized. It was not uncommon
for fresh checkouts to fail due to floating dependencies, or for things to break
when trying to switch to an older commit.
For day-to-day development, I think Bazel served us reasonably well - we could
generally switch between branches while being confident that builds would be
correct and reasonably fast, and not require full rebuilds (except on Windows,
where the lack of a sandbox and the TS rules would cause build breakages when TS
files were renamed/removed).
Bazel achieves that reliability by defining rules for each programming language
that define how source files should be turned into outputs. For the rules to
work with Bazel's sandboxing approach, they often have to reimplement or
partially bypass the standard tools that each programming language provides. The
Rust rules call Rust's compiler directly for example, instead of using Cargo,
and the Python rules extract each PyPi package into a separate folder that gets
added to sys.path.
These separate language rules allow proper declaration of inputs and outputs,
and offer some advantages such as caching of build products and fine-grained
dependency installation. But they also bring some downsides:
- The rules don't always support use-cases/platforms that the standard language
tools do, meaning they need to be patched to be used. I've had to contribute a
number of patches to the Rust, Python and JS rules to unblock various issues.
- The dependencies we use with each language sometimes make assumptions that do
not hold in Bazel, meaning they either need to be pinned or patched, or the
language rules need to be adjusted to accommodate them.
I was hopeful that after the initial setup work, things would be relatively
smooth-sailing. Unfortunately, that has not proved to be the case. Things
frequently broke when dependencies or the language rules were updated, and I
began to get frustrated at the amount of Anki development time I was instead
spending on build system upkeep. It's now about 2 years since switching to
Bazel, and I think it's time to cut losses, and switch to something else that's
a better fit.
The new build system is based on a small build tool called Ninja, and some
custom Rust code in build/. This means that to build Anki, Bazel is no longer
required, but Ninja and Rust need to be installed on your system. Python and
Node toolchains are automatically downloaded like in Bazel.
This new build system should result in faster builds in some cases:
- Because we're using cargo to build now, Rust builds are able to take advantage
of pipelining and incremental debug builds, which we didn't have with Bazel.
It's also easier to override the default linker on Linux/macOS, which can
further improve speeds.
- External Rust crates are now built with opt=1, which improves performance
of debug builds.
- Esbuild is now used to transpile TypeScript, instead of invoking the TypeScript
compiler. This results in faster builds, by deferring typechecking to test/check
time, and by allowing more work to happen in parallel.
As an example of the differences, when testing with the mold linker on Linux,
adding a new message to tags.proto (which triggers a recompile of the bulk of
the Rust and TypeScript code) results in a compile that goes from about 22s on
Bazel to about 7s in the new system. With the standard linker, it's about 9s.
Some other changes of note:
- Our Rust workspace now uses cargo-hakari to ensure all packages agree on
available features, preventing unnecessary rebuilds.
- pylib/anki is now a PEP420 implicit namespace, avoiding the need to merge
source files and generated files into a single folder for running. By telling
VSCode about the extra search path, code completion now works with generated
files without needing to symlink them into the source folder.
- qt/aqt can't use PEP420 as it's difficult to get rid of aqt/__init__.py.
Instead, the generated files are now placed in a separate _aqt package that's
added to the path.
- ts/lib is now exposed as @tslib, so the source code and generated code can be
provided under the same namespace without a merging step.
- MyPy and PyLint are now invoked once for the entire codebase.
- dprint will be used to format TypeScript/json files in the future instead of
the slower prettier (currently turned off to avoid causing conflicts). It can
automatically defer to prettier when formatting Svelte files.
- svelte-check is now used for typechecking our Svelte code, which revealed a
few typing issues that went undetected with the old system.
- The Jest unit tests now work on Windows as well.
If you're upgrading from Bazel, updated usage instructions are in docs/development.md and docs/build.md. A summary of the changes:
- please remove node_modules and .bazel
- install rustup (https://rustup.rs/)
- install rsync if not already installed (on windows, use pacman - see docs/windows.md)
- install Ninja (unzip from https://github.com/ninja-build/ninja/releases/tag/v1.11.1 and
place on your path, or from your distro/homebrew if it's 1.10+)
- update .vscode/settings.json from .vscode.dist
