Commit graph

49 commits

Author SHA1 Message Date
Damien Elmes
25c041bc0f unquote does need utf8, but we need to convert back to unicode after 2014-08-05 12:59:31 +09:00
Damien Elmes
08510a4a53 Revert "we shouldn't encode to utf8 when unquoting"
This reverts commit 2dd28d86a2.
2014-08-05 12:58:14 +09:00
Damien Elmes
2dd28d86a2 we shouldn't encode to utf8 when unquoting 2014-08-04 12:54:54 +09:00
Damien Elmes
d53346d783 limit url unquoting to image tags
this prevents random text like %20 in a field from being converted
when note is saved
2014-08-01 09:42:28 +09:00
Damien Elmes
dd2b6cb07d ignore >100MB files 2014-07-30 04:32:18 +09:00
Damien Elmes
cf801e4fb4 display more feedback when syncing media
deletes in particular take some time for the server to process, but
don't require much bandwidth, leading to the progress appearing to
have pause when content is actually being processed

this also gives the user an idea of how long the process will take to
complete
2014-07-28 17:00:26 +09:00
Damien Elmes
f54f5870fa deauthing should reset lastUsn to 0
thanks to houssam for the heads up!
2014-07-20 03:37:31 +09:00
Damien Elmes
3224a155a7 normalize filenames as we sync
user still needs to run "check media" if their fields are encoded
incorrectly, but by fixing on the fly we'll ensure mediaSanity doesn't
fail
2014-07-15 11:49:03 +09:00
Damien Elmes
d3807e3180 don't redetect delete which leads to sync error 2014-07-08 13:45:29 +09:00
Damien Elmes
999c53b5ec fix another downgrade+upgrade bug 2014-07-05 11:37:28 +09:00
Damien Elmes
1cb2aa077a must make sure to detach old media db before renaming 2014-07-05 07:21:32 +09:00
Damien Elmes
014787c4cb log mediaChangesZip() 2014-07-04 18:14:45 +09:00
Damien Elmes
eba3aa583c if media.db import fails, start anew 2014-06-28 05:06:43 +09:00
dae
2aa7714f87 new media syncing protocol
- media syncing no longer locks the account, so it can be done
  in the background in the future, and multiple clients can safely
  sync media at the same time
- all operations are now idempotent, so they can be repeatedly safely
  in the event of a connection error
- whether it's a normal incremental sync, an initial sync,
  or the media database has been deleted, no files will be uploaded
  or downloaded if they already exist on the other side
- file removals are now chunked like additions & updates, preventing
  timeouts due to large requests
- if the server can't process a chunk in time, it will return a count
  of what it did process, so the client can retry the rest

Notes for AnkiDroid:

- when porting this, recommend you pick a different name for the
.media.db2 file, so users don't accidentally copy the AD version to
the desktop or vice versa
- please make sure filenames are added to the zip in NFC form
2014-06-26 09:00:38 +09:00
Damien Elmes
de8adfecff disallow \r and \n in media filenames 2014-03-06 09:00:47 +09:00
Damien Elmes
f0525a76fb if we renamed any files to nfc, need to rerun check 2013-12-13 20:24:39 +09:00
Damien Elmes
54e2d42fa6 off by one in zip count comparison 2013-12-09 17:37:43 +09:00
Damien Elmes
ba589ac8cc need to compare mediaList with nfd on macs
otherwise the server says we have file x and local code
doesn't notice it already exists
2013-12-04 11:43:57 +09:00
Damien Elmes
1c35a590e3 fix regression in unused media check 2013-11-29 02:07:31 +09:00
Damien Elmes
0372f30220 adjust media regexp to not trigger on mce_src
otherwise pasting the following will cause an error:

<img src="..." mce_src="/foo.png">
2013-11-27 19:24:41 +09:00
Damien Elmes
99d82c1f2d fix hasIllegal check, and associated unit test 2013-11-26 03:19:11 +09:00
Damien Elmes
f6b9dadf13 catch invalid file encodings in media check & sync 2013-11-13 17:19:25 +09:00
Damien Elmes
5f0ef85044 fix reversed check in media 2013-10-30 21:10:25 +09:00
Damien Elmes
82a54c780f catch invalid temp folder and other tweaks
- tweak sync code so that a failure in loading the collection
  won't leave the app with an unopen collection
- don't show corrupt collection message when the error is not
  a db error
- catch the temp folder issue when loading the collection. i suspect
  this was the issue that was causing some people to end up with
  an open anki instance with no collection loaded
2013-10-22 15:30:53 +09:00
Damien Elmes
da1672d42f don't rename/unlink files for non-local check 2013-10-17 17:59:26 +09:00
Damien Elmes
6be7503543 normalize to nfc when sending media in sync 2013-10-16 08:37:34 +09:00
Damien Elmes
b42c0c725b Revert "tolerate a str arg to writeData()"
This reverts commit 2d4e88afbd.

On second thought, this change is likely to mask the error in most cases, and
we want add-on authors to update their add-ons anyway as they would not be
handling non-latin text in 2.0.12.
2013-10-10 11:08:39 +09:00
Damien Elmes
2d4e88afbd tolerate a str arg to writeData() 2013-10-09 13:13:48 +09:00
Damien Elmes
6c304ebe2f to normalize unicode we need to have unicode 2013-10-06 04:53:11 +09:00
Damien Elmes
0d1d8c5bf9 define standard encoding for unicode (#893)
- always store media references in fields in NFC form
- always encode filenames on disk in NFC form on machines other than macs
- use relevant encoding when placing files in the media folder during syncs
  and apkg imports as well
- rename 'unused media' back to 'check media'
- check media can now automatically change media references and filename
  encodings to the correct format
2013-09-20 19:28:44 +09:00
Damien Elmes
a538e29480 refactor illegal char handling and file adding for media
- add writeData() so we can add files to media folder without having to
  save them in an intermediate folder first
- we were stripping or checking for illegal filename characters in
  multiple places; factor those checks out into separate routines
- add * to invalid char list, and disallow both forward and backslash
- remove size checks in syncAdd(); is handled server-side
2013-07-11 15:33:24 +09:00
Damien Elmes
f2d538d546 catch errors trying to determine fs type 2013-06-14 13:15:05 +09:00
Damien Elmes
a6b44cff01 skip fat32 check on unmapped network drive (#902) 2013-06-10 15:32:21 +09:00
Damien Elmes
ec888eb2aa fix bug in media sync where we weren't fetching needed files 2013-05-31 14:03:34 +09:00
Damien Elmes
9714143af9 remove dropbox image escape hack 2013-05-29 09:34:12 +09:00
Damien Elmes
dceacff47c tell server which files we need 2013-05-27 15:29:24 +09:00
Damien Elmes
0a81df60dc automatically resync media, and don't send media server already has (#583) 2013-05-27 14:25:30 +09:00
Damien Elmes
97a342ae23 when media deleted during sync, move to trash instead of deleting 2013-05-23 13:38:51 +09:00
Damien Elmes
918694a096 more media regexp fixes
- split quoted and unquoted image case into separate regexps, since we can't
  include a group reference inside a character set
- disallow spaces in the non-quoted case
- this should fix matching on images with other attributes again
2013-05-22 09:45:58 +09:00
Damien Elmes
85cb7ef47e ignore _ in field references (#803) 2013-05-18 18:57:24 +09:00
Damien Elmes
4aa634bc44 need greedy match on images 2013-05-17 16:20:27 +09:00
Damien Elmes
8f36841281 ensure we match on identical closing quote (#769) 2013-05-17 12:23:56 +09:00
Damien Elmes
13a2943667 full media scan on fat32 drives (#844) 2013-05-16 17:58:30 +09:00
Damien Elmes
c8f4d3a582 check checksum of all filename alternatives when adding media
patch thanks to Julien Baley
2013-05-16 17:24:20 +09:00
ospalh
d806f44484 Only the Mac file system stores file names NFD-normalized. So do that normalization only there when looking for files. 2013-04-11 17:37:06 +09:00
Damien Elmes
f3f7835032 add upper bound to # of media files in single zip 2013-04-11 14:13:11 +09:00
Damien Elmes
bf1af126b0 properly generate clozed latex in unused media check (#775) 2013-04-11 13:26:52 +09:00
Damien Elmes
b2457811a2 optimize imports 2012-12-22 09:17:10 +09:00
Damien Elmes
d477df8c28 merged libanki and ankiqt into single distro 2012-12-21 16:51:59 +09:00