Commit graph

313 commits

Author SHA1 Message Date
Damien Elmes
e99a7c0f90 tweak naming and move method into col.decks 2020-12-20 10:26:16 +10:00
k12ish
cc0572a385 Added type hints, renamed kwarg 2020-12-19 18:12:58 +00:00
k12ish
ef1f58c8b6 Added method 2020-12-19 17:59:07 +00:00
Damien Elmes
9d1b6231d7 merge pylib references 2020-11-17 19:23:06 +10:00
Arthur Milchior
7850d2d662 NF: uses consts in col 2020-09-27 09:01:47 +02:00
Damien Elmes
56ceb6ba76 set_deck() 2020-09-03 17:42:46 +10:00
Damien Elmes
ce49ca9401 log manual reschedule, but ignore the log entry in the stats 2020-09-02 17:56:23 +10:00
Damien Elmes
d3dede057a move bury/suspend into backend 2020-09-01 10:24:38 +10:00
Damien Elmes
603210149c update to latest isort, pylint and pytest 2020-08-31 12:05:36 +10:00
Damien Elmes
d266f9a80a revlog entry should not be deleted when undoing preview card
The plan is to add a revlog entry when previewing in the future; this
is just a temporary fix.
2020-08-19 16:44:06 +10:00
Arthur Milchior
452daf8d80 Remove useless variables 2020-07-17 08:38:48 +02:00
Damien Elmes
d99ea0c7a0 fix media not working after full sync 2020-06-24 09:52:02 +10:00
Damien Elmes
83b3d3249f drop yellow colour in "filtered" for now
It's difficult to read against a light background
2020-06-16 16:39:07 +10:00
Damien Elmes
b51f03085e migrate card stats to backend
Currently this renders the HTML directly like the previous Python
implementation - doing it in JS would probably make more sense in the
future.
2020-06-15 17:22:16 +10:00
Damien Elmes
772c7a945c fix lint issues with latest mypy_protobuf 2020-06-14 15:38:43 +10:00
Damien Elmes
4a69b55a90 add note/card removal to backend 2020-06-04 18:21:04 +10:00
Damien Elmes
0e5b7da62a login/full up/full down plugged in 2020-06-01 13:57:10 +10:00
Damien Elmes
ee6d7f82e7 rework progress handling
- client now polls status instead of backend pushing it
- supports multiple threads
- update throttling happens in one place
2020-06-01 13:57:10 +10:00
evandrocoan
ef5c38dbc6 Added super().__repr__() to new __repr__() calls 2020-05-31 19:39:19 -03:00
evandrocoan
1e216e47ed Added __repr__ functions to common objects 2020-05-31 19:39:19 -03:00
Damien Elmes
97618564f4 fix typechecking breaking with latest mypy_protobuf
the change that caused it:
https://github.com/dropbox/mypy-protobuf/issues/118

This is more awkward to handle now, as the types are only available
at type-checking time. Python's static typing is such a mess :-(
2020-05-27 09:14:02 +10:00
Damien Elmes
38508c3ad7 use keyword args for calls with more than one argument 2020-05-24 09:12:47 +10:00
Damien Elmes
89dde3aeb0 migrate the remaining methods 2020-05-24 08:36:50 +10:00
Damien Elmes
4bf8175bcb migrate more scheduling/media/etc
almost there
2020-05-23 21:34:19 +10:00
Damien Elmes
081a61a438 more methods 2020-05-23 14:01:36 +10:00
Damien Elmes
175afa9fee migrate more methods to service 2020-05-22 22:09:33 +10:00
Damien Elmes
50fdf9b03d storage->collection 2020-05-20 17:58:28 +10:00
Damien Elmes
c49c378296 move storage logic into collection.py; fix export bug
https://anki.tenderapp.com/discussions/ankidesktop/41495-using-file-export-closes-the-collection-on-mwcoldb-if-the-browser-window-is-open
2020-05-20 17:43:34 +10:00
Damien Elmes
b9b837e7bd update before_upload() 2020-05-17 14:13:21 +10:00
Damien Elmes
7c5980a941 <= in modified_after_begin for unit tests 2020-05-15 14:24:59 +10:00
Damien Elmes
a2b7a30841 fetch/set remaining collection properties as required 2020-05-15 13:33:37 +10:00
Damien Elmes
c601dcef24 remove obsolete preview code 2020-05-14 09:22:53 +10:00
Damien Elmes
826cbb0108 fetch template and note fields in backend during normal card render
Saves having to serialize the note fields and q/a templates, which
is particularly a win when rendering question/answer in the browse
screen.

Also some work towards being able to preview notes without having to
commit them to the database.
2020-05-13 10:38:49 +10:00
Damien Elmes
9317cee9ba handle scheduling preferences in the backend 2020-05-12 21:13:34 +10:00
Damien Elmes
eee0d7e92f switch server back into a bool and rely on config 2020-05-12 21:13:34 +10:00
Damien Elmes
2c7900989c fix default rollover 2020-05-12 21:13:34 +10:00
Damien Elmes
70cc1699a6 rewrite DB check
- notes with wrong field count are now recovered instead of
being deleted
- notes with missing note types are now recovered
- notes with missing cards are now recovered
- recover_missing_deck() still needs implementing
- checks required
2020-05-12 21:13:34 +10:00
Damien Elmes
2413f286b1 bulk tag add/remove/update; canonify on note save
also remove the tag list updated hook - we'll need a better solution in
the future than having the library code call back into the GUI code
2020-05-12 21:13:34 +10:00
Damien Elmes
7bab99d873 support disabling unicode normalization in notes 2020-05-12 21:13:34 +10:00
Damien Elmes
a7a485d550 use backend for genCards() and updateFieldCache() 2020-05-12 21:13:34 +10:00
Damien Elmes
05ca797ee6 add pep8 name for findReplace 2020-05-12 21:13:34 +10:00
Damien Elmes
5fb5338d97 add missing decks in backend
- need to compare parents with unicode case folding
- duplicate check enforced by the DB
2020-05-12 21:13:33 +10:00
Damien Elmes
f592672fa9 add separate decks table, and start on moving deck handling to Rust
The Python tests are passing, but there are still a number of issues
to work through, and the table/protobuf schema is not yet finalized.
2020-05-12 21:13:33 +10:00
Damien Elmes
6e1d2990a0 remove unused _updateRequired and associated code 2020-05-12 21:13:33 +10:00
Damien Elmes
6e8860cafa hook the empty cards code up to the GUI 2020-05-12 21:13:33 +10:00
Damien Elmes
f637ac957d hook up new note and note type handling
- notetypes are fetched from the DB as needed, and cached in Python
- handle note type changes in the backend. Multiple operations can now
be performed in one go, but this is not currently exposed in the GUI.
- extra methods to grab sorted note type names quickly, and fetch by
name
- col.models.save() without a provided notetype is now a no-op
- note loading/saving handled in the backend
- notes with no valid cards can now be added
- templates can now be deleted even if they would previously
orphan notes

a number of fixmes have been left in notes.py and models.py
2020-05-12 21:13:33 +10:00
Damien Elmes
fcb8567185 drop legacy newBury option 2020-05-12 21:13:33 +10:00
Damien Elmes
c02716ccd8 python formatting+lints 2020-05-08 17:30:27 +10:00
Damien Elmes
a095a8bb07 fix due of preview cards being reset
https://anki.tenderapp.com/discussions/ankidesktop/41243-undo-function-causes-error-in-filtered-decks
2020-05-08 15:25:54 +10:00
Damien Elmes
54734ec88e fix error undoing card in preview queue
https://anki.tenderapp.com/discussions/ankidesktop/41243-undo-function-causes-error-in-filtered-decks
2020-05-08 15:25:38 +10:00
Damien Elmes
e37774f0bd fix v1 log appearing next to export 2020-05-05 16:15:49 +10:00
Damien Elmes
9bb4a02e0a fix unnecessary spaces being added to notes with empty tags 2020-05-04 21:53:22 +10:00
Damien Elmes
ff170bff31 fix tags with missing leading/trailing spaces in DB check 2020-04-30 08:54:17 +10:00
Damien Elmes
683f664d85 fix deleting deck configs 2020-04-20 13:23:05 +10:00
Damien Elmes
2f1523f82a set db mod flag if mtime changed by backend 2020-04-11 15:41:21 +10:00
Damien Elmes
4c7210b430 (de)serialize decks in backend 2020-04-09 12:41:59 +10:00
Damien Elmes
36ec7830a9 load/save note types in backend
This allows us to normalize bad data, and is the first step towards
splitting note types into separate tables.
2020-04-08 10:05:07 +10:00
Damien Elmes
676f4e74a8 store config in separate DB table
- mtime is tracked on each key individually, which will allow
merging of config changes when syncing in the future
- added col.(get|set|remove)_config()
- in order to support existing code that was mutating returned
values (eg col.conf["something"]["another"] = 5), the returned list/dict
will be automatically wrapped so that when the value is dropped, it
will save the mutated item back to the DB if it's changed. Code that
is fetching lists/dicts from the config like so:

col.conf["foo"]["bar"] = baz
col.setMod()

will continue to work in most case, but should be gradually updated to:

conf = col.get_config("foo")
conf["bar"] = baz
col.set_config("foo", conf)
2020-04-06 15:39:47 +10:00
Damien Elmes
7375a0389a add flag to skip downgrade on collection close
Disabled for now; when enabled it will allow faster collection
open and close in the normal case, while continuing to downgrade
when exporting or doing a full sync.

Also, when downgrading is disabled, the journal mode is no longer
changed back to delete.
2020-04-04 17:21:45 +10:00
Damien Elmes
ac4284b2de update tag handling
- tag list stored in a separate DB table
- non-wildcard searches now do full unicode case folding
(eg tag:masse matches 'Maße')
- wildcard matches do simple unicode case folding
- some functions haven't been updated yet, so ascii folding will
continue to be used in some operations
2020-04-03 19:34:46 +10:00
Damien Elmes
ab6eeb5882 ensure pending deck/tag/note type changes flushed before searching 2020-04-03 19:34:46 +10:00
Damien Elmes
8c80f96314 high due numbers shouldn't force a full sync 2020-03-26 09:09:11 +10:00
Damien Elmes
11a4d582b4 convert asc to desc instead of appending desc to the end of the order
as the latter doesn't work when sorting on more than one column

https://anki.tenderapp.com/discussions/beta-testing/1868-anki-2124-beta#comment_48174812
2020-03-23 19:53:57 +10:00
Damien Elmes
4e2e0d1b84 fix setting of wal 2020-03-23 13:52:57 +10:00
Damien Elmes
cd9ceebd59 simplify how the local offset is passed around
- no need to store it in conf
- move local_minutes_west() call to collection
2020-03-23 13:52:52 +10:00
Damien Elmes
f28e57a367 add enum for controlling sort order
eg col.find_cards("", order=BuiltinSortKind.CARD_DUE)
2020-03-22 12:59:24 +10:00
Damien Elmes
a5787781d7 add note searching 2020-03-21 09:00:05 +10:00
Damien Elmes
5debd3e0f8 add the ability to provide a custom sort order; use backend for find 2020-03-21 07:55:21 +10:00
Damien Elmes
d7daa63dbd make sure we set db to None so we can tell when we've closed the DB 2020-03-20 21:15:23 +10:00
Damien Elmes
874bc085fe support opening and closing the DB while keeping backend alive
This is safer than just dropping the backend, as .close() will
block if something else is holding the mutex. Also means we can
drop the extra I18nBackend code.

Media syncing still needs fixing.
2020-03-20 21:15:23 +10:00
Damien Elmes
7986a79530 remove db.close() 2020-03-20 21:15:23 +10:00
Damien Elmes
fa12213e98 move .reopen() to mw; fix exporting 2020-03-20 21:15:23 +10:00
Damien Elmes
90de4a267d drop lock() and setAutocommit()
We no longer need to worry about pysqlite implicitly beginning
transactions, and can be more explicit about beginning/ending
transactions

save() now also has a trx argument controlling whether a
transaction should be started / left open
2020-03-20 21:15:23 +10:00
Damien Elmes
ae06b9e446 add Collection struct, and get media check working again
- media check no longer needs collection to be closed
- use savepoints for operations initiated by Rust, so they are
atomic without forcing a commit
2020-03-20 21:15:23 +10:00
Damien Elmes
2cd7885ec0 add begin/commit/rollback, and support creating collections
all but one unit test is now passing
2020-03-20 21:15:23 +10:00
Damien Elmes
6db4418f05 drop log= argument from Collection 2020-03-20 21:15:23 +10:00
Damien Elmes
b876d97770 use (or)json for DB bridge
Some initial testing with orjson indicates performance varies from
slightly better than pysqlite to about 2x slower depending on the type
of query.

Performance could be improved by building the Python list in rspy
instead of sending back json that needs to be decoded, but it may make
more sense to rewrite the hotspots in Rust instead. More testing is
required in any case.
2020-03-20 21:15:23 +10:00
Damien Elmes
5778459d7c drop .cursor() 2020-03-20 21:15:23 +10:00
Damien Elmes
c1252d68f0 clone db.py into dbproxy.py 2020-03-20 21:15:23 +10:00
Damien Elmes
862e2b48f0 add col.weakref() and fix more reference cycles in pylib 2020-03-05 09:47:47 +10:00
Damien Elmes
6b204274c0 fix retain cycle caused by col.tr
We can't use a weakref.proxy with a method, so manually proxy the
call instead.
2020-03-05 09:45:51 +10:00
Alan Du
6c2dda6c9c Monkeytype qt/aqt/reviewer.py 2020-03-01 10:16:08 -05:00
Damien Elmes
8960d12aac add nfd tag fix to db check
we already normalize in the editor and importing, so perhaps these
tags were from an old version

https://anki.tenderapp.com/discussions/ankidesktop/39120-bug-filter-do-not-work-with-tags-which-contain-german-letters-like
2020-03-01 11:21:25 +10:00
Damien Elmes
43f5d13ed6 migrate strings with trailing newlines to fluent 2020-02-27 20:36:40 +10:00
Alan Du
cbeb47f8df Improve mypy config
- Make mypy stricter
- Warn for unnecessary workarounds
2020-02-26 22:22:40 -05:00
Damien Elmes
4d917cc65b add self.col.tr() shortcut 2020-02-27 12:33:15 +10:00
Damien Elmes
f900f24f60 more changes to the template code
- _renderQA() has moved to template.py:render_card()
- dropped QAData in favour of a properly typed dict
- render_card() returns a TemplateRenderOutput struct instead of a dict
- card_did_render now takes that output as the first arg, and can
mutate it
- TemplateRenderContext now stores the original card, so it can return
a card even in the add screen case

The old mungeFields and mungeQA hook have been removed as part of this
change. mungeQA can be replaced with the card_did_render hook.
In the mungeFields case, please switch to using field_filter instead.
2020-01-24 13:44:13 +10:00
Damien Elmes
836213e587 renderQA() wasn't used in the codebase 2020-01-24 11:09:38 +10:00
Damien Elmes
17ebdfc099 extract and flag AV tags as part of the render process
We can now show replay buttons for the audio contained in {{FrontSide}}
without having to play it again when the answer is shown.

The template code now always defers FrontSide rendering, as it wasn't
a big saving, and meant the logic had to be implemented twice.
2020-01-24 11:06:11 +10:00
Damien Elmes
12c60f20fe _localOffsetForDate() was broken
It was including the elapsed time of day when calculating
the offset, leading to incorrect results
2020-01-17 18:52:36 -07:00
Damien Elmes
d3cc63efb0 move types.py into the files responsible for each type 2020-01-17 10:17:33 +10:00
Damien Elmes
2a0b480103 add context to card rendering, and rework related hooks
- the context exists for the lifecycle of one card's render,
and caches calls to things like .card() to avoid add-ons needing to
do their own cache management.
- add-ons can optionally add extra data to the context if they need
it across multiple filters
- removed card_will_render. the legacy hook is still available for
now
- card_did_render is now called only once, with both front and back
text
2020-01-17 09:30:42 +10:00
Damien Elmes
d2865235df i18n 2020-01-16 21:05:29 +10:00
Damien Elmes
912a49a2e3 fix empty cloze message not appearing 2020-01-16 18:47:21 +10:00
Damien Elmes
b56c9591c0 more useful template error message 2020-01-16 18:47:21 +10:00
Damien Elmes
cc9a36c11a simplify previewCards() and use existing card ids where possible
The type arg is no longer used, as neither type 0 nor 1 appears to
have been used in the codebase.

By using the existing card ids, it allows add-ons that gather
information about a card to work properly in the card template screen
without extra hacks.
2020-01-16 09:19:41 +10:00
Damien Elmes
b09e7e8247 more tweaks for readability/consistency 2020-01-15 17:45:35 +10:00
Damien Elmes
237c0791f8 more hook tweaks, and merge the pre-render field and template hooks 2020-01-15 17:18:11 +10:00
Damien Elmes
cab572b63c remove _hook/_filter suffix 2020-01-15 16:53:24 +10:00
Damien Elmes
b2f756f1b7 tweak the hook names in anki/
still a work in progress
2020-01-15 16:43:22 +10:00
Damien Elmes
2fb5ff5d9c fix broken answer displaying 2020-01-14 09:00:52 +10:00
Damien Elmes
662eb53e6a switch to classes for hooks
This allows us to add a docstring to .append() so users can see
the names of the arguments that are being passed, and means we
don't have to remember to prepend run_ when calling a hook.
2020-01-14 08:54:07 +10:00
Damien Elmes
33bf4c204b add hook to modify template prior to rendering 2020-01-13 21:57:15 +10:00
Damien Elmes
b86ae31907 update the rest of the anki/ hooks/filters 2020-01-13 17:59:52 +10:00
Damien Elmes
dd61389319 New type-safe approach to hooks/filters
Still todo:
- Add separate module for GUI hooks
- Update the remaining runHook/runFilter() calls
- Document the changes, including defensive registration
2020-01-13 13:57:51 +10:00
Damien Elmes
9bb0348fdd more template rendering tweaks
- The front and back are rendered in one call now. If the front
side contains no custom filters, we can bake {{FrontSide}} into the
rear side. If it did contain custom filters, we return the partially
complete rear template instead, and the calling code can inject
the FrontSide in after it has been fully rendered.

- Instead of modifying "cloze" into something like "cq-2", the card
ordinal and whether we're rendering the question or answer are now
passed in to the rendering filters as context.

- The Rust code doesn't need to support filter names split on '-'
anymore.

- Drop the "Show" part of hint descriptions so i18n support can be
deferred.

- Ignore blank filter names caused by user using two colons instead
of one.

- Fixed hint field and text transposition.
2020-01-12 21:34:07 +10:00
Damien Elmes
a51a4e4d31 drop pystache and move legacy code into separate file 2020-01-11 19:38:30 +10:00
Damien Elmes
a5613523ee rework new timezone code
We now store the UTC offset that was in effect at creation time,
and use that to determine the starting date.
2020-01-09 16:58:27 +10:00
Damien Elmes
0087eee6d9 handle conditional replacement in Rust
This extends the existing Rust code to handle conditional
replacement. The replacement of field names and filters to text
remains in Python, so that add-ons can still define their own
field modifiers.

The code is currently running the old Pystache rendering and the
new implementation in parallel, and will print a message to the
console if they don't match. If you notice any problems, please
let me know.
2020-01-08 20:28:04 +10:00
Damien Elmes
1f2e00690f move the rest of Anki's code out of pystache
In the process of factoring out the field filtering, the "extra"
and "fullname" args are just passed in as a blank string now.
Extra was functionality that allowed a field modifier to be defined
as "filtername(arg1,arg2):field", and fullname was the name of the
field including any provided field modifiers. From grepping through
the add-ons on AnkiWeb, neither appears to have been used.
2020-01-08 17:15:46 +10:00
Damien Elmes
4bca26161b clean up _renderQA(), and split rendering part out 2020-01-08 12:07:44 +10:00
Damien Elmes
99141d9dfb add a partial Python implementation of the backend 2020-01-06 12:24:47 +10:00
Damien Elmes
5876866565 tweaking the folder names again
hopefully that's the last of it
2020-01-03 07:48:38 +10:00
Renamed from lib-python/anki/collection.py (Browse further)