Commit graph

82 commits

Author SHA1 Message Date
RumovZ
872b6df22a
Optimise searching in (all) fields (#1622)
* Avoid rebuilding regex in field search

* Special case search in all fields

* Don't repeat mid nodes in field search sql

Small speed gain for searches like `*:re:foo` and reduces the sql tree
depth if a lot of field names of the same notetype match.

* Add sql function to match fields with regex

* Optimise used field search algorithm

- Searching in all fields is a special case.
- Using native SQL comparison is preferred.
- For Regex, use newly added SQL function.

* Please clippy

* Avoid pyramid of doom

* nt_fields -> matched_fields

* Add tests for regex and all field searches

* minor tweaks for readability (dae)
2022-01-24 20:30:08 +10:00
Damien Elmes
ca0374782e update to latest rusqlite 2021-06-25 16:22:21 +10:00
Damien Elmes
50961a9196 push review randomizing into SQL
This makes the review backlog case more expensive, since we end up
shuffling items outside the daily limit, but for the common case it's
about the same speed, and it means we don't need two separate sorting
steps. New cards remain handled the same way, since a backlog
is common there.

Also ensures that interday learning cards honor the deck sorting, and
that the non-default sort orders shuffle at the end.
2021-06-01 14:50:35 +10:00
Damien Elmes
d878de54c0 default to the v2 scheduler in new collections
ported from 53d9433d94
2021-05-06 19:09:28 +10:00
Damien Elmes
64ebc32b3d tidy up Rust imports
rustfmt can do this automatically, but only when run with a nightly
toolchain, so it needs to be manually done for now - see rslib/rusfmt.toml
2021-04-18 18:38:54 +10:00
Damien Elmes
262b50445c start on making deck config and schema/mod changes undoable
+ move timestamps into a struct in a separate file for convenience
2021-04-18 17:33:12 +10:00
Damien Elmes
76eb119870 add schema change prompt to removal, tweak return struct 2021-04-18 17:33:12 +10:00
Damien Elmes
a250464309 switch DbError to tuple type 2021-04-01 17:34:03 +10:00
Damien Elmes
094e4ad461 crate::err -> crate::error 2021-04-01 16:07:13 +10:00
Damien Elmes
1055acb9f2 fix more issues uncovered by the latest clippy 2021-03-27 20:25:34 +10:00
Damien Elmes
dc81a7fed0 use mixed case for abbreviations in Rust code
So, this is fun. Apparently "DeckId" is considered preferable to the
"DeckID" were were using until now, and the latest clippy will start
warning about it. We could of course disable the warning, but probably
better to bite the bullet and switch to the naming that's generally
considered best.
2021-03-27 19:53:33 +10:00
Damien Elmes
3433c02242 i18n->tr in rslib/ to match Python/TS code 2021-03-27 12:09:51 +10:00
Damien Elmes
8fc43956c2 move collection mtime bump into backend
Fixes the following issue:
- some code directly modifies the database, causing modified_in_python
to be set to true
- an undoable operation is run, which calls autosave() at the end
- autosave() notices there's an undoable operation, and commits immediately
- because modified_in_python was true, col.mtime was bumped in Python
- that invalidated the undo queue, preventing the operation from being
undone
2021-03-19 19:45:21 +10:00
Damien Elmes
c0e939ceb6 split config.rs up 2021-03-10 11:53:27 +10:00
Damien Elmes
2c6b6734b5 experimental queue building
Still a work in progress, and hidden behind a feature flag.
2021-03-01 12:18:21 +10:00
Damien Elmes
6a44269280 sched->scheduler 2021-02-23 17:35:20 +10:00
Damien Elmes
53d9433d94 Revert "use v2 scheduler+new timezone handling by default for new users"
This reverts commit 8372931b9b.

I fear this will be too disruptive - let's give AnkiDroid a bit more
time to catch up. Reverting this will mean new users are presented with
an upgrade notice on first startup, which looks a bit silly, but it's
probably the lesser of two evils.
2021-02-21 19:03:46 +10:00
Damien Elmes
8372931b9b use v2 scheduler+new timezone handling by default for new users
- In corner cases, enabling the new timezone handling later can cause
reviews to shift forward or back a day, so it's best to have it on
by default.
- https://github.com/ankidroid/Anki-Android/issues/5805 has not landed
in a stable release yet, but will hopefully not be too far off by the
time 2.1.41 is released.
- Existing users will be unaffected, as the upgrade prompt in the previous
commit asks them if they use AnkiDroid.
- Users starting on AnkiDroid will be unaffected, as their collections
will still be on V1.
- The error message AnkiWeb gives when syncing an older AnkiDroid
with the new timezone enabled has been updated to direct users to the
preferences screen.
2021-02-21 17:18:08 +10:00
Damien Elmes
09dfa9ced6 allow normal sync tests to run offline 2021-01-12 18:47:08 +10:00
Damien Elmes
90fa38495e reindex when checking database
Came across a user with a corrupt index:

sqlite> pragma integrity_check;
 integrity_check = wrong # of entries in index ix_revlog_cid
 integrity_check = wrong # of entries in index ix_cards_sched

This is not picked up by a quick check, and a vacuum does not
fix it, but a reindex does.

AnkiWeb currently performs a full check when a file is uploaded
to it, so this was leading to "corrupt" to show up when syncing
the collection, with a local DB check not reporting/fixing the issue.
2020-10-05 17:40:43 +10:00
Damien Elmes
76acf04dc0 update schema to fix default initial ease in deck configs
Closes #766

- changes the on-disk representation from % to a multiplier,
eg 250 -> 2.5, as this is consistent with the other options
- resets deck configs at or below 1.3 to 2.5
- for any cards that were using a reset deck config, reset their
current factor if it's at or below 2.0x. The cutoff is arbitrary,
and just intended to make sure we catch cards the user has rated
Easy on multiple times. The existing due dates are left alone.
2020-09-22 08:16:39 +10:00
Damien Elmes
5574c0dfb3 set schema on collection creation 2020-07-03 11:55:27 +10:00
Damien Elmes
263bfb2e8e on upgrade, mark decks modified that are renamed 2020-06-14 14:48:58 +10:00
Damien Elmes
4d7e23111e change sync label to indicate sync state
- blue for normal sync, red for full sync required
- refactor status fetching code so we don't hold a collection lock
during the network request, which slows things down
- fix sync spinner restarting when returning to deck list
2020-06-02 13:23:01 +10:00
Damien Elmes
529152aef6 rework usn handling in sync 2020-06-01 13:57:10 +10:00
Damien Elmes
f10d0ee0cb more syncing work
no checks yet
2020-06-01 13:57:10 +10:00
Damien Elmes
b9b837e7bd update before_upload() 2020-05-17 14:13:21 +10:00
Damien Elmes
da43d22aa5 update rusqlite 2020-05-16 19:49:08 +10:00
Damien Elmes
9317cee9ba handle scheduling preferences in the backend 2020-05-12 21:13:34 +10:00
Damien Elmes
2c7900989c fix default rollover 2020-05-12 21:13:34 +10:00
Damien Elmes
5ec7251f6e stop using memory for sqlite temp store
On large collections this can end up consuming hundreds of megabytes
of RAM on operations like a vacuum.
2020-05-12 21:13:34 +10:00
Damien Elmes
70cc1699a6 rewrite DB check
- notes with wrong field count are now recovered instead of
being deleted
- notes with missing note types are now recovered
- notes with missing cards are now recovered
- recover_missing_deck() still needs implementing
- checks required
2020-05-12 21:13:34 +10:00
Damien Elmes
2413f286b1 bulk tag add/remove/update; canonify on note save
also remove the tag list updated hook - we'll need a better solution in
the future than having the library code call back into the GUI code
2020-05-12 21:13:34 +10:00
Damien Elmes
f592672fa9 add separate decks table, and start on moving deck handling to Rust
The Python tests are passing, but there are still a number of issues
to work through, and the table/protobuf schema is not yet finalized.
2020-05-12 21:13:33 +10:00
Damien Elmes
b89dd32f78 mod schema if field/templates changed
Instead of throwing an error if schema not marked as changed,
just mark it changed, as that way it can be included as part of
the same transaction.
2020-05-12 21:13:33 +10:00
Damien Elmes
e8f4a75bd5 always fetch full notetype 2020-05-12 21:13:33 +10:00
Damien Elmes
8cfb1fad44 use native struct for NoteType 2020-05-12 21:13:33 +10:00
Damien Elmes
540892639f add stock note types in backend 2020-05-12 21:13:33 +10:00
Damien Elmes
805a3a710e split note types into separate tables
- store the config in protobuf instead of json
- still loading+saving in bulk for now
- code using the schema11 structs needs to be migrated
2020-05-12 21:13:33 +10:00
Damien Elmes
4c7210b430 (de)serialize decks in backend 2020-04-09 12:41:59 +10:00
Damien Elmes
36ec7830a9 load/save note types in backend
This allows us to normalize bad data, and is the first step towards
splitting note types into separate tables.
2020-04-08 10:05:07 +10:00
Damien Elmes
34cca119e3 use blobs for config 2020-04-06 15:39:47 +10:00
Damien Elmes
676f4e74a8 store config in separate DB table
- mtime is tracked on each key individually, which will allow
merging of config changes when syncing in the future
- added col.(get|set|remove)_config()
- in order to support existing code that was mutating returned
values (eg col.conf["something"]["another"] = 5), the returned list/dict
will be automatically wrapped so that when the value is dropped, it
will save the mutated item back to the DB if it's changed. Code that
is fetching lists/dicts from the config like so:

col.conf["foo"]["bar"] = baz
col.setMod()

will continue to work in most case, but should be gradually updated to:

conf = col.get_config("foo")
conf["bar"] = baz
col.set_config("foo", conf)
2020-04-06 15:39:47 +10:00
Damien Elmes
7375a0389a add flag to skip downgrade on collection close
Disabled for now; when enabled it will allow faster collection
open and close in the normal case, while continuing to downgrade
when exporting or doing a full sync.

Also, when downgrading is disabled, the journal mode is no longer
changed back to delete.
2020-04-04 17:21:45 +10:00
Damien Elmes
45ea0ddefd use separate schema version for tag changes
Will prevent issues if user upgrades after an unclean shutdown
2020-04-04 10:14:51 +10:00
Damien Elmes
ac4284b2de update tag handling
- tag list stored in a separate DB table
- non-wildcard searches now do full unicode case folding
(eg tag:masse matches 'Maße')
- wildcard matches do simple unicode case folding
- some functions haven't been updated yet, so ascii folding will
continue to be used in some operations
2020-04-03 19:34:46 +10:00
Damien Elmes
dfa1ce6429 translate default deck config name 2020-04-01 17:49:25 +10:00
Damien Elmes
35c03af520 split deck config into separate SQL table
- on collection load, the schema is upgraded to 12
- on collection close, the changes are reversed so older clients
can continue to open the collection
- in the future, we could potentially skip the reversal except
when exporting/doing a full sync
- the same approach should work for decks, note types and tags in the
future too
- the deck list code needs updating to cache the deck confs for the
life of the call
2020-03-30 20:01:16 +10:00
Damien Elmes
004cc2b5f8 move deck conf handling to backend 2020-03-30 14:39:46 +10:00
Damien Elmes
bf83715ee0 initial work on undo support 2020-03-29 17:52:16 +10:00