Commit graph

263 commits

Author SHA1 Message Date
Damien Elmes
14263c7a79 move protobuf into separate folder in preparation for multiple files 2021-07-09 21:02:40 +10:00
Damien Elmes
7c27c4b9a0 move backend.proto into rslib/ 2020-11-02 16:28:31 +10:00
Damien Elmes
ce3b08ac58 initial Bazel conversion
Running and testing should be working on the three platforms, but
there's still a fair bit that needs to be done:

- Wheel building + testing in a venv still needs to be implemented.
- Python requirements still need to be compiled with piptool and pinned;
need to compile on all platforms then merge
- Cargo deps in cargo/ and rslib/ need to be cleaned up, and ideally
unified into one place
- Currently using rustls to work around openssl compilation issues
on Linux, but this will break corporate proxies with custom SSL
authorities; need to conditionally use openssl or use
https://github.com/seanmonstar/reqwest/pull/1058
- Makefiles and docs still need cleaning up
- It may make sense to reparent ts/* to the top level, as we don't
nest the other modules under a specific language.
- rspy and pylib must always be updated in lock-step, so merging
rspy into pylib as a private module would simplify things.
- Merging desktop-ftl and mobile-ftl into the core ftl would make
managing and updating translations easier.
- Obsolete scripts need removing.
- And probably more.
2020-11-01 14:26:58 +10:00
Damien Elmes
f32d773e25 set_deck() 2020-09-03 17:42:46 +10:00
Damien Elmes
7c169daa95 move card sorting and resetting to backend 2020-09-03 15:54:15 +10:00
Damien Elmes
366c5c4774 log manual reschedule, but ignore the log entry in the stats 2020-09-02 17:56:23 +10:00
Damien Elmes
31b7d7d645 move reschedCards() to backend 2020-09-02 17:56:23 +10:00
Damien Elmes
396081d329 move filtered deck empty/fill to backend
emptying of individual card ids still to be done
2020-09-02 17:56:23 +10:00
Damien Elmes
7f42500e53 rename some card+note fields in backend 2020-09-01 10:24:38 +10:00
Damien Elmes
9f51347e1e move bury/suspend into backend 2020-09-01 10:24:38 +10:00
Damien Elmes
988b3c00c1 move unbury/unsuspend routines into backend 2020-09-01 10:24:38 +10:00
Damien Elmes
999b659c3a reimplement congrats screen in Rust+Typescript 2020-09-01 10:24:38 +10:00
Damien Elmes
47d3666ad0 fix hour graph not handling timezones west of UTC
https://forums.ankiweb.net/t/statistics-bugs-after-update-2-1-5-to-2-1-29/1620
2020-07-31 13:33:41 +10:00
Damien Elmes
42ec6e0a18 fix early reviews not appearing in graph
https://forums.ankiweb.net/t/differences-between-new-and-old-stats-2-1-28/1602
2020-07-31 13:33:41 +10:00
Damien Elmes
3a75161778 fix deck tree with a day delta, and support arbitrary timestamps 2020-07-06 19:16:03 +10:00
Damien Elmes
693fcfce7d i18n card counts 2020-06-29 15:48:01 +10:00
Damien Elmes
349eca843b get i18n working in typescript 2020-06-29 15:48:01 +10:00
Damien Elmes
0293f2220c qt's js code now shares ts/node_modules; more graph work 2020-06-29 15:48:01 +10:00
Damien Elmes
4d88b6521d cards added graph 2020-06-29 15:48:01 +10:00
Damien Elmes
1e9901e9bd experiment with exposing raw card/revlog data to frontend 2020-06-29 15:48:00 +10:00
Damien Elmes
fd258d15c5 some initial work on updating the graphs 2020-06-29 15:48:00 +10:00
Damien Elmes
df8be64da3 protobuf package should be in PascalCase 2020-06-29 15:48:00 +10:00
Damien Elmes
f5694a7e4a migrate card stats to backend
Currently this renders the HTML directly like the previous Python
implementation - doing it in JS would probably make more sense in the
future.
2020-06-15 17:22:16 +10:00
Damien Elmes
75471224fa add progress to db check 2020-06-08 21:07:36 +10:00
Damien Elmes
e74e5c604b add daily count updating to backend 2020-06-05 19:49:53 +10:00
Damien Elmes
55f8b87548 support generating a due tree for a different date 2020-06-05 09:38:31 +10:00
Damien Elmes
c5e65d02e1 add note/card removal to backend 2020-06-04 18:21:04 +10:00
Damien Elmes
d7a2e78c0b change sync label to indicate sync state
- blue for normal sync, red for full sync required
- refactor status fetching code so we don't hold a collection lock
during the network request, which slows things down
- fix sync spinner restarting when returning to deck list
2020-06-02 13:23:01 +10:00
Damien Elmes
ad3082d1c8 handle aborting normal sync
- Use a separate abort handle, as the media sync is running
in the background and we need to be able to target them separately.
The current progress handling is going to need a rethink if we introduce
any other background tasks in the future.
- Roll back the transaction when interrupting.
2020-06-01 13:57:10 +10:00
Damien Elmes
da1e0527ca report normal sync progress
Also:
- provide a way for the progress handler to skip the throttling so that
we can ensure progress is updated at the end of a stage
- show 'checking' at the end of full sync
2020-06-01 13:57:10 +10:00
Damien Elmes
0787381a99 login/full up/full down plugged in 2020-06-01 13:57:10 +10:00
Damien Elmes
aaed0464e5 rework progress handling
- client now polls status instead of backend pushing it
- supports multiple threads
- update throttling happens in one place
2020-06-01 13:57:10 +10:00
Damien Elmes
c4032c0176 another snapshot of sync work
progress handler is going to need to be reworked to function across
threads
2020-06-01 13:57:10 +10:00
Damien Elmes
2114fdf6f5 more syncing work
no checks yet
2020-06-01 13:57:10 +10:00
Damien Elmes
91e3256e6a move dupe check to backend 2020-05-24 19:48:56 +10:00
Damien Elmes
e9fea26d1c use keyword args for calls with more than one argument 2020-05-24 09:12:47 +10:00
Damien Elmes
bbcc566e6d migrate the remaining methods 2020-05-24 08:36:50 +10:00
Damien Elmes
93f18c14d4 migrate more scheduling/media/etc
almost there
2020-05-23 21:34:19 +10:00
Damien Elmes
8b93a0c19d migrate notetypes, update GIL list 2020-05-23 20:43:55 +10:00
Damien Elmes
35489d5fa7 add some more newtypes to the RPC defs 2020-05-23 16:58:01 +10:00
Damien Elmes
8df2ab65fb migrate cards and notes 2020-05-23 16:19:48 +10:00
Damien Elmes
32a071cf79 migrate decks and dconf methods 2020-05-23 15:09:16 +10:00
Damien Elmes
927ed25b02 more methods 2020-05-23 14:01:36 +10:00
Damien Elmes
d57ded706f migrate more methods to service 2020-05-22 22:09:33 +10:00
Damien Elmes
36024ed086 start reworking protobuf handling
Will allow us to cut down on boilerplate by automatically generating
code from RPC service definitions
2020-05-22 20:56:15 +10:00
Damien Elmes
4438be2c8d protobuf tidyups 2020-05-18 09:26:14 +10:00
Damien Elmes
a682865169 move deck config to protobuf 2020-05-17 13:28:41 +10:00
Damien Elmes
e85911c4e4 add option to limit deck tree counts to a particular node 2020-05-17 08:38:49 +10:00
Damien Elmes
c60229779c switch to new deck tree in deck browser
Saves us having to look up collapsed/filtered as we render, and gives
us type completion.
2020-05-16 10:52:14 +10:00
Damien Elmes
cbfc9d5a87 handle default deck and filtered deck suppression in the backend 2020-05-15 21:21:10 +10:00
Damien Elmes
dc6471ad40 fetch stock notetypes from backend 2020-05-15 17:08:24 +10:00
Damien Elmes
6dd31341f8 add "fill empty" checkbox 2020-05-14 20:58:45 +10:00
Damien Elmes
4a00ad075c drop availOrds(), and use backend for getting cloze numbers 2020-05-14 12:14:00 +10:00
Damien Elmes
257e893d95 update the card layout screen
- changes are now committed in bulk when closing the dialog,
and can be canceled
- it's not necessary to save the note to the database to preview it
- duplicate fields are now shown as duplicates in the top list
- redraw preview more quickly
- use + instead of _ when deduplicating names, as the latter is a
glob character
2020-05-13 17:24:49 +10:00
Damien Elmes
14c0a5891b fetch template and note fields in backend during normal card render
Saves having to serialize the note fields and q/a templates, which
is particularly a win when rendering question/answer in the browse
screen.

Also some work towards being able to preview notes without having to
commit them to the database.
2020-05-13 10:38:49 +10:00
Damien Elmes
21ac2c14a7 handle scheduling preferences in the backend 2020-05-12 21:13:34 +10:00
Damien Elmes
2eaa759478 fetch timing_today() params in backend 2020-05-12 21:13:34 +10:00
Damien Elmes
1018204e4f switch server back into a bool and rely on config 2020-05-12 21:13:34 +10:00
Damien Elmes
d7f9bccd1e rewrite DB check
- notes with wrong field count are now recovered instead of
being deleted
- notes with missing note types are now recovered
- notes with missing cards are now recovered
- recover_missing_deck() still needs implementing
- checks required
2020-05-12 21:13:34 +10:00
Damien Elmes
667f9e6554 bulk tag add/remove/update; canonify on note save
also remove the tag list updated hook - we'll need a better solution in
the future than having the library code call back into the GUI code
2020-05-12 21:13:34 +10:00
Damien Elmes
85528cabc5 use backend for genCards() and updateFieldCache() 2020-05-12 21:13:34 +10:00
Damien Elmes
ad1c519d1c move find&replace to backend 2020-05-12 21:13:34 +10:00
Damien Elmes
7ef72aba04 use the backend for the deck due tree
- approx 3x faster on a large test deck
- counts are no longer capped to 1000 in the tree
2020-05-12 21:13:33 +10:00
Damien Elmes
267be1b050 add missing decks in backend
- need to compare parents with unicode case folding
- duplicate check enforced by the DB
2020-05-12 21:13:33 +10:00
Damien Elmes
97e857a41c add browserCollapsed as part of tree build 2020-05-12 21:13:33 +10:00
Damien Elmes
13d23cd24b speed up browser load by rendering deck tree in Rust and skipping counts 2020-05-12 21:13:33 +10:00
Damien Elmes
24ef0cc6f1 add separate decks table, and start on moving deck handling to Rust
The Python tests are passing, but there are still a number of issues
to work through, and the table/protobuf schema is not yet finalized.
2020-05-12 21:13:33 +10:00
Damien Elmes
2a17f99eaa remove unused _updateRequired and associated code 2020-05-12 21:13:33 +10:00
Damien Elmes
21d7df9473 empty card handling 2020-05-12 21:13:33 +10:00
Damien Elmes
c8cbc13bba start on exposing notes and individual note type methods
changes to note:

- add_note() now takes a provided deck id instead of looking it up
in the notetype
- note type use counts fetched using a single table scan
- make sure note type changes are persisted
- expose optionalness of ords in templates and fields json
2020-05-12 21:13:33 +10:00
Damien Elmes
d47e83c989 embed enum to avoid conflicts 2020-05-12 21:13:33 +10:00
Damien Elmes
3807dcacd5 native struct for CardTemplate 2020-05-12 21:13:33 +10:00
Damien Elmes
91767fc87b native struct for NoteField
ord is made optional so we can keep track of changes in the future
2020-05-12 21:13:33 +10:00
Damien Elmes
4566a35085 expose stock notetypes 2020-05-12 21:13:33 +10:00
Damien Elmes
65f12eeda2 add stock note types in backend 2020-05-12 21:13:33 +10:00
Damien Elmes
475478cec5 split note types into separate tables
- store the config in protobuf instead of json
- still loading+saving in bulk for now
- code using the schema11 structs needs to be migrated
2020-05-12 21:13:33 +10:00
Damien Elmes
f5f2782e3b embed sort kind enum to avoid future namespace conflicts 2020-04-14 20:31:10 +10:00
Damien Elmes
0538d9b764 (de)serialize decks in backend 2020-04-09 12:41:59 +10:00
Damien Elmes
bca49bd054 load/save note types in backend
This allows us to normalize bad data, and is the first step towards
splitting note types into separate tables.
2020-04-08 10:05:07 +10:00
Damien Elmes
02d0d9a670 add separate json error 2020-04-06 16:29:31 +10:00
Damien Elmes
913c5b9854 use bytes for deck config proto 2020-04-06 15:39:47 +10:00
Damien Elmes
6471adafcc use bytes for config proto 2020-04-06 15:39:47 +10:00
Damien Elmes
b5022ad354 store config in separate DB table
- mtime is tracked on each key individually, which will allow
merging of config changes when syncing in the future
- added col.(get|set|remove)_config()
- in order to support existing code that was mutating returned
values (eg col.conf["something"]["another"] = 5), the returned list/dict
will be automatically wrapped so that when the value is dropped, it
will save the mutated item back to the DB if it's changed. Code that
is fetching lists/dicts from the config like so:

col.conf["foo"]["bar"] = baz
col.setMod()

will continue to work in most case, but should be gradually updated to:

conf = col.get_config("foo")
conf["bar"] = baz
col.set_config("foo", conf)
2020-04-06 15:39:47 +10:00
Damien Elmes
ac54f85840 update tag handling
- tag list stored in a separate DB table
- non-wildcard searches now do full unicode case folding
(eg tag:masse matches 'Maße')
- wildcard matches do simple unicode case folding
- some functions haven't been updated yet, so ascii folding will
continue to be used in some operations
2020-04-03 19:34:46 +10:00
Damien Elmes
07c414692e preserve mtime/usn when syncing deck config, and add snake_case names 2020-04-03 19:34:46 +10:00
Damien Elmes
284ee0860b allow aborting a media sync while http req in flight
The progress handling code needs a rethink, as we now have two separate
ways to flag that the media sync should abort. In the future, it may
make sense to switch to polling the backend for progress, instead of
passing a callback in.
2020-04-01 09:49:25 +10:00
Damien Elmes
cffe146397 split deck config into separate SQL table
- on collection load, the schema is upgraded to 12
- on collection close, the changes are reversed so older clients
can continue to open the collection
- in the future, we could potentially skip the reversal except
when exporting/doing a full sync
- the same approach should work for decks, note types and tags in the
future too
- the deck list code needs updating to cache the deck confs for the
life of the call
2020-03-30 20:01:16 +10:00
Damien Elmes
089a68eb43 move deck conf handling to backend 2020-03-30 14:39:46 +10:00
Damien Elmes
fc846ac4e4 use add_card() 2020-03-27 15:11:07 +10:00
Damien Elmes
e28fdde984 handle mtime/usn bump in backend, and tweak integer sizes 2020-03-26 20:55:04 +10:00
Damien Elmes
0510ab7c9c update_card() 2020-03-26 18:54:20 +10:00
Damien Elmes
b2fba4e1ff use backend to get card 2020-03-26 17:47:43 +10:00
Damien Elmes
e02f032a47 use backend for v1 and v2 cutoff calculations
this should also fix the CI failures, which were happening because
the datetime module wasn't matching the shifted time.time()
2020-03-22 14:28:26 +10:00
Damien Elmes
b8aaa8fba4 add enum for controlling sort order
eg col.find_cards("", order=BuiltinSortKind.CARD_DUE)
2020-03-22 12:59:24 +10:00
Damien Elmes
0089f0149a add note searching 2020-03-21 09:00:05 +10:00
Damien Elmes
23717089e9 add the ability to provide a custom sort order; use backend for find 2020-03-21 07:55:21 +10:00
Damien Elmes
f36857ae27 prototype of integration
no ordering yet, and no tests
2020-03-20 21:15:23 +10:00
Damien Elmes
e5f883d1c3 support opening and closing the DB while keeping backend alive
This is safer than just dropping the backend, as .close() will
block if something else is holding the mutex. Also means we can
drop the extra I18nBackend code.

Media syncing still needs fixing.
2020-03-20 21:15:23 +10:00
Damien Elmes
3cacc00fc3 make the collection optional 2020-03-20 21:15:23 +10:00
Damien Elmes
bd7ded18c3 drop the protobuf prototype 2020-03-20 21:15:23 +10:00
Damien Elmes
a129b8f8cf initial work on DB handling in Rust
committing the Protobuf implementation for posterity, but will replace
it with json, as Protobuf measures about 6x slower for some workloads
like 'select * from notes'
2020-03-20 21:15:23 +10:00
Damien Elmes
e512199ace add restore media action 2020-03-10 13:35:09 +10:00
Damien Elmes
cac9ccfa0d add empty trash action in aqt 2020-03-10 12:49:40 +10:00
Damien Elmes
67a101add1 add missing newline to .proto 2020-03-09 18:58:49 +10:00
Damien Elmes
c97b37b7ae make logging explicit, and support logging to a file 2020-02-29 20:44:01 +10:00
Damien Elmes
b030f2b40b info is superfluous 2020-02-27 10:53:49 +10:00
Damien Elmes
83d894ac87 return a localized error for all error kinds
some errors are not yet localized, but now the Python code
doesn't need to think about which property to use
2020-02-27 10:46:24 +10:00
Damien Elmes
94953cf6b5 separate out infallible and fallible commands in .proto 2020-02-27 08:46:09 +10:00
Damien Elmes
59328d9ade tweak rounding
- avoid rounding minutes
- round the seconds taken in the card info screen
- provide different precise and imprecise modes, since we need
to display seconds to multiple decimals in some areas
2020-02-25 17:29:06 +10:00
Damien Elmes
ea22296d4c expand clozes before rendering LaTeX
https://anki.tenderapp.com/discussions/ankidesktop/39071-latex-media-windows-version-2121-0cd1ee92-fail-rendering-latex
2020-02-24 18:34:08 +10:00
Damien Elmes
c98247fa0f translations no longer require an open collection 2020-02-23 14:57:02 +10:00
Damien Elmes
0d1a30c6d8 rework Fluent handling
- all .ftl files for a language are concatenated into a single file
at build time
- all languages are included in the binary
- external ftl files placed in the ftl folder can override the
built-in definitions
- constants are automatically generated for each string key
- dropped the separate StringsGroup enum
2020-02-23 13:22:50 +10:00
Damien Elmes
72528ee41c add deck config category 2020-02-21 20:53:20 +10:00
Damien Elmes
e84fa7dee3 add next learning due time + remaining count to congrats screen
https://anki.tenderapp.com/discussions/ankidesktop/38000-v2-scheduler-learning-cards
2020-02-21 19:54:50 +10:00
Damien Elmes
3a83d9844a add studied_today(), move to statistics.ftl 2020-02-21 18:01:15 +10:00
Damien Elmes
74bff450e2 add non-abbreviated timespan translation; update existing short=True calls
- drop the '5m3s' special casing done in the card stats screen, and
just use decimals
- change alignment of the review log so that the non-abbreviated
spans are easier to read
2020-02-21 15:29:38 +10:00
Damien Elmes
11647daec1 move answer_button_time to the backend, split sched into separate module 2020-02-21 15:29:38 +10:00
Damien Elmes
2633d6db4b move answer button labels into fluent 2020-02-21 15:29:38 +10:00
Damien Elmes
cf1824dac5 update to latest fluent-rs and add basic locale-aware decimals
- git version pinned at the moment until the concurrency fix
lands in 0.10.2
- currently float values are hard-coded at 2 decimal places;
we should switch to using NUMBER() in the future
2020-02-21 15:29:38 +10:00
Damien Elmes
fd01cac4d7 split "Due" into three different contexts for translators 2020-02-17 14:41:01 +10:00
Damien Elmes
f40a00f0cb localize some error messages 2020-02-17 11:38:22 +10:00
Damien Elmes
4b63a07c99 expose translations to Python 2020-02-17 08:40:17 +10:00
Damien Elmes
1f736239cd pass progress back as translated string(s) 2020-02-17 08:40:17 +10:00
Damien Elmes
78ffde32e1 expose StringsGroup enum in protobuf 2020-02-17 08:40:17 +10:00
Damien Elmes
0738ff46b6 make template errors translatable 2020-02-17 08:40:17 +10:00
Damien Elmes
834989a019 include report in MediaCheckOutput 2020-02-17 08:40:17 +10:00
Damien Elmes
36d2536262 add i18n support to Rust code using Fluent 2020-02-17 08:40:17 +10:00
Damien Elmes
1383973753 shorten add_file_to_media_folder 2020-02-17 08:40:17 +10:00
Damien Elmes
89626c1a72 use media.trash for unused media deletion as well 2020-02-17 08:40:17 +10:00
Damien Elmes
d32e2fe6eb add extract_latex to backend; use it for latex build 2020-02-17 08:40:17 +10:00
Damien Elmes
790f3c9f72 plug new media check in 2020-02-17 08:40:17 +10:00
Damien Elmes
edf60b10d4 backend stores media folder/db locations; don't need to pass them in 2020-02-17 08:40:17 +10:00
Damien Elmes
9c60f2c77e log media DB change registration progress, and allow aborting it 2020-02-17 08:40:17 +10:00
Damien Elmes
ef14b1a96b simplify code by accumulating stats at the source 2020-02-17 08:40:17 +10:00
Damien Elmes
1f931a5b78 defer errors on upload to media check 2020-02-17 08:40:17 +10:00
Damien Elmes
77158760ff use enums for some common errors 2020-02-17 08:40:16 +10:00
Damien Elmes
738e415463 connect media sync progress to Python, add a progress dialog 2020-02-17 08:40:16 +10:00
Damien Elmes
50d8a40e23 add progress hook to media sync 2020-02-17 08:40:16 +10:00
Damien Elmes
78bfbef863 media sync working, but unpolished 2020-02-17 08:40:16 +10:00
Damien Elmes
1acb3b86b9 backend init can now fail, and update media db when file is added
- Adding files inside Anki now updates the media DB, so a full file
scan at sync time is no longer required if no other changes have been
made.
- Use a protobuf message for backend initialization, and return a
string error if initialization fails.
2020-02-17 08:40:16 +10:00
Damien Elmes
9bc38647e6 initial rs implementation of media database 2020-02-17 08:40:16 +10:00
Damien Elmes
c9a5ed3cc6 add add_file() and write_data() 2020-02-17 08:40:16 +10:00
Damien Elmes
391a58077a show which side the template error occurred on 2020-02-08 11:54:08 +10:00
Damien Elmes
86a580d820 reuse reveal_cloze_text() for LaTeX cloze expansion 2020-01-28 07:40:44 +10:00
Damien Elmes
47bcb3845c support speed control in tts tags 2020-01-26 14:31:07 +10:00
Damien Elmes
46c6a7f7e4 extract and flag AV tags as part of the render process
We can now show replay buttons for the audio contained in {{FrontSide}}
without having to play it again when the answer is shown.

The template code now always defers FrontSide rendering, as it wasn't
a big saving, and meant the logic had to be implemented twice.
2020-01-24 11:06:11 +10:00
Damien Elmes
2ee8902bd9 added flag_av_tags() for replay button support 2020-01-21 14:44:27 +10:00
Damien Elmes
b0d2958721 add rank handling to TTS; parse TTS args in get_av_tags() 2020-01-21 12:41:37 +10:00
Damien Elmes
548ed31ef7 add a simple TTS field filter
This simply wraps the field in extra text that the frontend will
deal with. Also added some helpers for extracting and stripping
audio and TTS tags from the rendered text.
2020-01-20 19:16:24 +10:00
Damien Elmes
c525a75fa3 drop plus_one test 2020-01-17 18:54:38 -07:00