- moved tags into json like previous changes, and dropped the unnecessary id
- added tags.py for a tag manager
- moved the tag utilities from utils into tags.py
Like the previous change, models have been moved from a separate DB table to
an entry in the deck. We need them for many operations including reviewing,
and it's easier to keep them in memory than half on disk with a cache that
gets cleared every time we .reset(). This means they are easily serialized as
well - previously they were part Python and part JSON, which made access
confusing.
Because the data is all pulled from JSON now, the instance methods have been
moved to the model registry. Eg:
model.addField(...) -> deck.models.addField(model, ...).
- IDs are now timestamped as with groups et al.
- The data field for plugins was also removed. Config info can be added to
deck.conf; larger data should be stored externally.
- Upgrading needs to be updated for the new model structure.
- HexifyID() now accepts strings as well, as our IDs get converted to strings
in the serialization process.
Rather than use a combination of id lookups on the groups table and a group
configuration cache in the scheduler, I've moved the groups and group config
into json objects on the deck table. This results in a net saving of code and
saves one or more DB lookups on each card answer, in exchange for a small
increase in deck load/save work.
I did a quick survey of AnkiWeb, and the vast majority of decks use less than
100 tags, and it's safe to assume groups will follow a similar pattern.
All groups and group configs except the default one will use integer
timestamps now, to simplify merging when syncing and importing.
defaultGroup() has been removed in favour of keeping the models up to date
(not yet done).
- should never skip recording graves, for the sake of merging
- 1.0 upgrade will fail on decks that have the same fact creation date. need
to work around this in the future
The approach of using incrementing id numbers works for syncing if we assume
the server is canonical and all other clients rewrite their ids as necessary,
but upon reflection it is not sufficient for merging decks in general, as we
have no way of knowing whether objects with the same id are actually the same
or not. So we need some way of uniquely identifying the object.
One approach would be to go back to Anki 1.0's random 64bit numbers, but as
outlined in a previous commit such large numbers can't be handled easy in some
languages like Javascript, and they tend to be fragmented on disk which
impacts performance. It's much better if we can keep content added at the same
time in the same place on disk, so that operations like syncing which are mainly
interested in newly added content can run faster.
Another approach is to add a separate column containing the unique id, which
is what Mnemosyne 2.0 will be doing. Unfortunately it means adding an index
for that column, leading to slower inserts and larger deck files. And if the
current sequential ids are kept, a bunch of code needs to be kept to ensure ids
don't conflict when merging.
To address the above, the plan is to use a millisecond timestamp as the id.
This ensures disk order reflects creation order, allows us to merge the id and
crt columns, avoids the need for a separate index, and saves us from worrying
about rewriting ids. There is of course a small chance that the objects to be
merged were created at exactly the same time, but this is extremely unlikely.
This commit changes models. Other objects will follow.
the initial plan was to zero the creation time and leave the cards/facts there
until we have a chance to garbage collect them on a schema change, but such an
approach won't work with deck subscriptions
- use negative numbers to denote second intervals
- record the rev ivl when leaving lrn queue
- improve revlog upgrade
- don't truncate precision when recording time taken
reps should now be equal to the number of entries in the revlog, and only
exists so that we can order by review count in the browser efficiently
streak is no longer necessary as we have a learn queue now
Previously cloze deletions were handled by copying the contents of one field
into another and applying transforms to it. This had a number of problems:
- after you add a card, you can't undo the cloze deletion
- if you spot a mistake, you have to edit it twice (or more if you have more
than one cloze for a sentence)
- making multiple clozes requires copying & pasting the sentence multiple
times
- this also lead to much bigger decks if the sentences being cloze-deleted are
large
- related clozes can't be spaced apart as siblings
To address these issues, we introduce the idea of cloze tags in the card
template and fields. If the template has the text:
{{cloze:1:field}}
And a field has the following contents:
{{c1::hello}}
Then the template will automatically replace that part of the text with either
occluded text, or a highlighted answer. All other clozes in the field are
displayed normally.
At the same time, we add support for text: into the template library, instead
of manually creating text: fields in the dict for every field.
Finally, add a forecast routine to get the due counts for the next week, which
is used in the GUI.
The media table was originally introduced when Anki hashed media filenames,
and needed a way to remember the original filename. It also helped with:
1) getting a quick list of all media used in the deck, or the media added
since the last sync, for mobile clients
2) merging identical files with different names
But had some drawbacks:
- every operation that modifies templates, models or facts meant generating
the q/a and checking if any new media had appeared
- each entry is about 70 bytes, and some decks have 100k+ media files
So we remove the media table. We address 1) by being more intelligent about
media downloads on the mobile platform. We ask the user after a full sync if
they want to look for missing media, and they can choose not to if they know
they haven't added any. And on a partial sync, we can scan the contents of the
incoming facts for media references, and download any references we find. This
also avoids all the issues people had with media not downloading because it
was in their media folder but not in the media database.
For 2), when copying media to the media folder, if we have a duplicate
filename, we check if that file has the same md5, and avoid copying if so.
This won't merge identical content that has separate names, but instances
where users need that are rare.
The 'entry due' is the due time of a failed card before it enters the learning
queue. When the card graduates or is removed, it has its old due time
restored. We could pull this from the revlog, but it's cheaper to do it this
way.
