Commit graph

51 commits

Author SHA1 Message Date
RumovZ
f182dc3c3a Backend colpkg exporting (#1719)
* Implement colpkg exporting on backend

* Use exporting logic in backup.rs

* Refactor exporting.rs

* Add backend function to export collection

* Refactor backend/collection.rs

* Use backend for colpkg exporting

* Don't use default zip compression for media

* Add exporting progress

* Refactor media file writing

* Write dummy collections

* Localize dummy collection note

* Minimize dummy db size

* Use `NamedTempFile::new()` instead of `new_in`

* Drop redundant v2 dummy collection

* COLLECTION_VERSION -> PACKAGE_VERSION

* Split `lock_collection()` into two to drop flag

* Expose new colpkg in GUI

* Improve dummy collection message

* Please type checker

* importing-colpkg-too-new -> exporting-...

* Compress the media map in the v3 package (dae)

On collections with lots of media, it can grow into megabytes.

Also return an error in extract_media_file_names(), instead of masking
it as an optional.

* Store media map as a vector in the v3 package (dae)

This compresses better (eg 280kb original, 100kb hashmap, 42kb vec)

In the colpkg import case we don't need random access. When importing
an apkg, we will need to be able to fetch file data for a given media
filename, but the existing map doesn't help us there, as we need
filename->index, not index->filename.

* Ensure folders in the media dir don't break the file mapping (dae)
2022-03-15 16:48:02 +10:00
gnnoh
d60a1a6b37 Fix ValueError when exported files have wrong mtime (#1666)
* Fix ValueError when exported files have wrong mtime

Set the `strict_timestamps` argument to `False`, so the media files which have a wrong mtime can be normally added to the zipfile.

* Update CONTRIBUTORS (ankitects#1666)

* Reformat exporting.py
2022-02-16 11:17:20 +10:00
RumovZ
7c2e0ab2d4 Remove redundant camelcase aliases (#1509) 2021-11-26 12:29:48 +10:00
RumovZ
fe514ddb30 PEP8 for rest of pylib (#1451)
* PEP8 dbproxy.py

* PEP8 errors.py

* PEP8 httpclient.py

* PEP8 lang.py

* PEP8 latex.py

* Add decorator to deprectate key words

* Make replacement for deprecated attribute optional

* Use new helper `_print_replacement_warning()`

* PEP8 media.py

* PEP8 rsbackend.py

* PEP8 sound.py

* PEP8 stdmodels.py

* PEP8 storage.py

* PEP8 sync.py

* PEP8 tags.py

* PEP8 template.py

* PEP8 types.py

* Fix DeprecatedNamesMixinForModule

The class methods need to be overridden with instance methods, so every
module has its own dicts.

* Use `# pylint: disable=invalid-name` instead of id

* PEP8 utils.py

* Only decorate `__getattr__` with `@no_type_check`

* Fix mypy issue with snakecase

Importing it from `anki._vendor` raises attribute errors.

* Format

* Remove inheritance of DeprecatedNamesMixin

There's almost no shared code now and overriding classmethods with
instance methods raises mypy issues.

* Fix traceback frames of deprecation warnings

* remove fn/TimedLog (dae)

Neither Anki nor add-ons appear to have been using it

* fix some issues with stringcase use (dae)

- the wheel was depending on the PyPI version instead of our vendored
version
- _vendor:stringcase should not have been listed in the anki py_library.
We already include the sources in py_srcs, and need to refer to them
directly. By listing _vendor:stringcase as well, we were making a
top-level stringcase library available, which would have only worked for
distributing because the wheel definition was also incorrect.
- mypy errors are what caused me to mistakenly add the above - they
were because the type: ignore at the top of stringcase.py was causing
mypy to completely ignore the file, so it was not aware of any attributes
it contained.
2021-10-25 14:50:13 +10:00
RumovZ
e37ccfdfa3 PEP8 pylib (#1443)
* PEP8 scheduler/base.py

* PEP8 _backend/__init__.py

* PEP8 _backend/genbackend.py

* PEP8 _backend/genfluent.py

* PEP8 scheduler/__init__.py

* PEP8 __init__.py

* PEP8 _legacy.py

* PEP8 syncserver/__init__.py

- Make 'ip' a good name
- Overrule `global col` being identified as a constant

* PEP8 syncserver/__main__.py

* PEP8 buildinfo.py

* Implement `DeprecatedNamesMixin` for modules

* PEP8 browser.py

* PEP8 config.py

* PEP8 consts.py

* PEP8 db.py

* Format

* Improve AttributeError for DeprecatedNamesMixin

* print the line that imported/referenced the legacy module attr (dae)

* DeprecatedNamesMixinStandalone -> ...ForModule
2021-10-22 20:39:49 +10:00
Damien Elmes
3c1729e91b run pyupgrade over codebase [python upgrade required]
This adds Python 3.9 and 3.10 typing syntax to files that import
attributions from __future___. Python 3.9 should be able to cope with
the 3.10 syntax, but Python 3.8 will no longer work.

On Windows/Mac, install the latest Python 3.9 version from python.org.
There are currently no orjson wheels for Python 3.10 on Windows/Mac,
which will break the build unless you have Rust installed separately.

On Linux, modern distros should have Python 3.9 available already. If
you're on an older distro, you'll need to build Python from source first.
2021-10-04 15:05:48 +10:00
Damien Elmes
f1c1826e84 fix mypy issue introduced in #1304 2021-07-21 18:00:53 +10:00
alex
464adc4aa8 fix card flag clearance when exporting 2021-07-19 22:14:52 +03:00
Damien Elmes
f9560586cb PEP8 collection.py 2021-06-27 15:12:22 +10:00
Damien Elmes
fba156ec93 PEP8 models.py 2021-06-27 14:30:00 +10:00
Damien Elmes
3c46c9a7eb PEP8 decks.py 2021-06-27 14:02:48 +10:00
Damien Elmes
d6e3964151 PEP8 cards.py 2021-06-27 12:12:23 +10:00
rgreenblatt
66c28ebb8f solve missing escape of ' (fixes #1144) 2021-04-22 00:54:11 -04:00
Damien Elmes
6b72aaf8c2 ID -> Id in protobuf and Python
follow-up to a90d5aa359
2021-03-27 21:38:20 +10:00
Damien Elmes
bd59e41f73 update some no-arg TR constants 2021-03-26 12:37:18 +10:00
Arthur Milchior
ea3b55a3a4 NF: DeckID type 2021-03-26 11:14:08 +10:00
Arthur Milchior
13710c1357 NF: CardID type 2021-03-26 11:14:08 +10:00
Damien Elmes
4d7c7e5b36 remove unnecessary setMod() calls 2021-03-10 11:47:53 +10:00
Damien Elmes
bf7528d90a minor code cleanups with pyupgrade
- pyupgrade --py38-plus --keep-runtime-typing --keep-percent-format
- third-party mpv and winpaths excluded
2021-02-11 09:43:40 +10:00
Damien Elmes
a6be0f493b start work on more clearly defining backend/protobuf boundaries
- anki._backend stores the protobuf files and rsbackend.py code
- pylib modules import protobuf messages directly from the
_pb2 files, and explicitly export any will be returned or consumed
by public pylib functions, so that calling code can import from pylib
- the "rsbackend" no longer imports and re-exports protobuf messages
- pylib can just consume them directly.
- move errors to errors.py

Still todo:

- rsbridge
- finishing the work on rsbackend, and check what we need to add
back to the original file location to avoid breaking add-ons
2021-01-31 18:55:45 +10:00
abdo
702c5712e4 Fix description of exporters
Description broke yet again with the full migration to Fluent
2020-11-19 23:52:46 +03:00
Damien Elmes
1781e83252 merge pylib references 2020-11-17 19:23:06 +10:00
Damien Elmes
f8d402ee70 fix error exporting v2 collection when translations don't match
https://forums.ankiweb.net/t/utf8-error-in-importing-deck/4430/13
2020-11-11 15:42:08 +10:00
ianki
28ad68d490 Add hooks for filtering media. 2020-11-09 02:12:23 -08:00
Damien Elmes
31f011b3a5 fix description in export screen (thanks to ANH)
https://forums.ankiweb.net/t/anki-2-1-31-exporting-display-bug/2669/3
2020-08-26 08:33:35 +10:00
Damien Elmes
1902e839e3 fix missing translations in export screen
https://forums.ankiweb.net/t/untranslated-strings/1623
2020-07-31 13:33:41 +10:00
Matt Krump
d4511e3287 Turn on check_untyped_defs for aqt.exporting 2020-07-28 18:42:22 -06:00
Damien Elmes
cafe5cbbc1 strip out \r on export
(private post)
https://anki.tenderapp.com/discussions/ankidesktop/42315-bug-report-exporting-selected-cards-in-browser-issue
2020-06-09 12:56:16 +10:00
Damien Elmes
068cc43095 storage->collection 2020-05-20 17:58:28 +10:00
Damien Elmes
8a42896fb8 speed up children() and add deck_and_child_ids() 2020-05-16 15:40:07 +10:00
Damien Elmes
8012d4ec69 fix exporting bug
can't/shouldn't pop up confirmation message on export
2020-05-15 11:03:34 +10:00
Damien Elmes
6350e18680 hook up new note and note type handling
- notetypes are fetched from the DB as needed, and cached in Python
- handle note type changes in the backend. Multiple operations can now
be performed in one go, but this is not currently exposed in the GUI.
- extra methods to grab sorted note type names quickly, and fetch by
name
- col.models.save() without a provided notetype is now a no-op
- note loading/saving handled in the backend
- notes with no valid cards can now be added
- templates can now be deleted even if they would previously
orphan notes

a number of fixmes have been left in notes.py and models.py
2020-05-12 21:13:33 +10:00
Damien Elmes
1834b65714 downgrade when writing dummy collection 2020-04-04 18:43:26 +10:00
Damien Elmes
3c7eabc51e add flag to skip downgrade on collection close
Disabled for now; when enabled it will allow faster collection
open and close in the normal case, while continuing to downgrade
when exporting or doing a full sync.

Also, when downgrading is disabled, the journal mode is no longer
changed back to delete.
2020-04-04 17:21:45 +10:00
Damien Elmes
07c414692e preserve mtime/usn when syncing deck config, and add snake_case names 2020-04-03 19:34:46 +10:00
Damien Elmes
258a8282d4 move .reopen() to mw; fix exporting 2020-03-20 21:15:23 +10:00
Damien Elmes
0dc4886558 add col.weakref() and fix more reference cycles in pylib 2020-03-05 09:47:47 +10:00
Damien Elmes
5e44fb0304 remove redundant declaration 2020-02-12 08:06:35 +10:00
Arthur Milchior
48339784cc deckIds return deck of selected cards 2020-02-11 10:50:59 -08:00
Arthur Milchior
0cace50f00 inverse if/else in deckIds 2020-02-11 10:50:59 -08:00
Arthur Milchior
ea0a509372 cardIds return cids if it is set 2020-02-11 10:50:59 -08:00
Arthur Milchior
17610d4c01 anki.exporter takes extra argument cids 2020-02-11 10:50:58 -08:00
Arthur Milchior
d1183df830 aqt.exporter takes extra argument cid 2020-02-11 10:49:13 -08:00
Arthur Milchior
538d5bd76a remove "typing.FOO" and import foo
According to grep, they were the only two instances
2020-02-04 08:49:53 -08:00
Arthur Milchior
c2f09df68c anki.exporting.AnkiExporter.deckIds
This change would help me simplify the code of
https://ankiweb.net/shared/info/1983204951

It would allow my exporter to use the code of AnkiExporter while
selecting which decks I export, instead of exporting all decks as I
was doing until today. (I never realized I was exporting all deck
actually, until someone sent me a bug report)
2020-02-03 01:36:27 -08:00
Damien Elmes
b22dc71810 more tweaks for readability/consistency 2020-01-15 17:45:35 +10:00
Damien Elmes
8bad40152d remove _hook/_filter suffix 2020-01-15 16:53:24 +10:00
Damien Elmes
a38b5c1bb6 tweak the hook names in anki/
still a work in progress
2020-01-15 16:43:22 +10:00
Damien Elmes
7352e0f43b switch to classes for hooks
This allows us to add a docstring to .append() so users can see
the names of the arguments that are being passed, and means we
don't have to remember to prepend run_ when calling a hook.
2020-01-14 08:54:07 +10:00
Damien Elmes
3d94af972e update the rest of the anki/ hooks/filters 2020-01-13 17:59:52 +10:00