Running and testing should be working on the three platforms, but
there's still a fair bit that needs to be done:
- Wheel building + testing in a venv still needs to be implemented.
- Python requirements still need to be compiled with piptool and pinned;
need to compile on all platforms then merge
- Cargo deps in cargo/ and rslib/ need to be cleaned up, and ideally
unified into one place
- Currently using rustls to work around openssl compilation issues
on Linux, but this will break corporate proxies with custom SSL
authorities; need to conditionally use openssl or use
https://github.com/seanmonstar/reqwest/pull/1058
- Makefiles and docs still need cleaning up
- It may make sense to reparent ts/* to the top level, as we don't
nest the other modules under a specific language.
- rspy and pylib must always be updated in lock-step, so merging
rspy into pylib as a private module would simplify things.
- Merging desktop-ftl and mobile-ftl into the core ftl would make
managing and updating translations easier.
- Obsolete scripts need removing.
- And probably more.
The previous implementation had some slightly questionable memory safety
properties (older versions of PyO3 didn't uphold the Rust aliasing rules
and would thus create multiple &mut references to #[pyclass] objects).
This explains why Backend has internal Mutex<T>s even though all of its
methods took &mut self.
The solution is to simply make all methods take &self, which luckily
doesn't pose too make issues -- most of the code inside Backend already
has sufficient locking. The only two things which needed to be
explicitly handled where:
1. "self.runtime" which was fairly easy to handle. All usages of
the Runtime only require an immutable reference to create a new
Handle, so we could switch to OnceCell which provides
lazy-initialisation semantics without needing a more heavy-handed
Mutex<tokio::runtime::Handle>.
2. "self.sync_abort" was simply wrapped in a Mutex<>, though some of the
odd semantics of sync_abort (not being able to handle multiple
processes synchronising at the same time) become pretty obvious with
this change (for now we just log a warning in that case). In
addition, switch to an RAII-style guard to make sure we don't forget
to clear the abort_handle.
As a result, we now no longer break Rust's aliasing rules and we can
build with newer versions of PyO3 which have runtime checks for these
things (and build on stable Rust).
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
- Use a separate abort handle, as the media sync is running
in the background and we need to be able to target them separately.
The current progress handling is going to need a rethink if we introduce
any other background tasks in the future.
- Roll back the transaction when interrupting.
Some initial testing with orjson indicates performance varies from
slightly better than pysqlite to about 2x slower depending on the type
of query.
Performance could be improved by building the Python list in rspy
instead of sending back json that needs to be decoded, but it may make
more sense to rewrite the hotspots in Rust instead. More testing is
required in any case.
- Adding files inside Anki now updates the media DB, so a full file
scan at sync time is no longer required if no other changes have been
made.
- Use a protobuf message for backend initialization, and return a
string error if initialization fails.
