SQLite Forum

sqlar : how to remove files

sqlar : how to remove files

(1) By Peter Etchells (peteretchn) on 2021-10-31 21:24:31 [link] [source]

sqlar looks a good alternative to other compression tools. But,

When I lazily type in

sqlite3 db.sqlar -Au *.*

the archive gets added to itself. Is this by design, or should I just be more careful?

Also, (triggered by the above), there does not seem to be a command to remove a file from the archive.

(2) By anonymous on 2021-11-01 04:33:38 in reply to 1 [link] [source]

You seem to be correct there isn't a command to remove a file from the archive, but can use the SQL DELETE command to delete files from an archive, by typing a SQL command at the sqlite> prompt, such as: delete from sqlar where name='db.sqlar';

(3) By Simon Slavin (slavin) on 2021-11-01 12:33:42 in reply to 2 [source]

Hmm. After doing this, if not adding more data back in, would one do VACUUM ?

This is an aspect of "everything must be a SQLite database' I've never considered before.

(4) By Warren Young (wyoung) on 2021-11-01 17:40:34 in reply to 1 [link] [source]

Is this by design

It's by mis-design, but the error is in your OS of choice, not in SQLite/SQLar.

You don't tell us outright that you're using Windows, but I can tell from your use of "*.*" that you are. Unix people tend not to use that wildcard because it means something different from a simple *, which is shorter and usually more correct besides.

This in turn tells me that you're using an OS where they decided that it's up to the called program to expand wildcards instead of leaving that to the shell. This has a number of bad side effects:

  1. Most immediately relevant to you, it means the expansion of *.* doesn't happen until the program is up and running and the sqlar DB has been created, so expanding that wildcard will include the just-created file. On Unix type systems, the equivalent expansion happens before the program's even run, so your symptom doesn't occur there.

  2. A lot of functionally duplicate code that doesn't behave the same way. Even in the case where you assume the arguments are expanded by the underlying C runtime library, a program's command line expansion may function differently depending on whether it was built with VC++, GCC, Clang... In the case of GCC, it may differ between native GCC, MinGW GCC, Cygwin GCC... Then there's the fact that programmers being programmers, some aren't satisfied with the default wildcard expansion and so write their own, leading to still more different ways wildcards are interpreted.

    You may guess that the plethora of Unix shells means the same problem happens on the other OSes, but it doesn't for two reasons. One, a given user tends to use just one shell, so they can get used to how their shell expands arguments in all cases. Two, POSIX sets minimal rules that all POSIX-type shells must obey, so they can't drift too wildly. You still get some oddities like whether wildcard expansion happens in the middle of a word or not (e.g. zsh vs bash) but goto "One".

  3. Programs not written in a language that has runtime support for glob expansion on program initialization (e.g. Windows batch files) have to do tricks to expand their args. It doesn't matter what language you write your program in on a Unix box: command line expansion is the shell's problem, so it always happens the same way, modulo the quibbles in point #2.

All of which is why I use a POSIX shell on Windows whenever possible. I get a consistent experience not just relative to macOS, Linux, BSD, etc. it means I get consistent behavior from one program to the next. The equivalent command

  $ sqlite3 foo.sqlar -Ac *

doesn't add foo.sqlar to the archive because it doesn't exist yet at command line expansion time.

there does not seem to be a command to remove a file from the archive.

That's true of the version built into the sqlite3 shell, but not of the standalone sqlar utility. It got that feature in 2016.

Perhaps that feature should be backported.

(5) By Larry Brasfield (larrybr) on 2021-11-01 17:53:30 in reply to 4 [link] [source]

Perhaps that feature should be backported.

The feature is going into the 3.37 release.

(8) By Warren Young (wyoung) on 2021-11-01 18:39:25 in reply to 5 [link] [source]

Thank you!

I was looking into the sqlar commit and noticed that it adds a use of GLOB for this case and was worried that it created a potential double-expansion problem until I realized why it does that: it only does it for -e, -l and -d, all cases where you're not dealing with files the shell can see.

Maybe this feature should use GLOB as well, for the same reason? How do I delete s* from a sqlar file otherwise, short of listing the archive and doing my own glob expansion, leading to problem #2 above?

(9) By Larry Brasfield (larrybr) on 2021-11-01 18:59:23 in reply to 8 [link] [source]

I considered using glob(x,y), and may have it in before the merge to trunk. Because feature freeze is imminent, I elected to defer that nicety in favor of leveraging existing code to process a known file list, without changing it. I expect to get globbing in later. Whether I can get it working and properly tested before v3.37 feature freeze is uncertain, so I want to be sure to get the basic function in first.

(11) By Larry Brasfield (larrybr) on 2021-11-02 00:53:50 in reply to 8 [link] [source]

Perhaps you or others interested can try breaking the archive_remove branch tip. Its name should be archive_remove_and_glob now. The archive content is subject to glob matching for --remove, --list and --extract subcommands. Glob matching for incoming files (--insert and --update) may come later.

(12) By Warren Young (wyoung) on 2021-11-02 03:22:02 in reply to 11 [link] [source]

Making glob an option has some merit, but I think following the lead of the standalone sqlar makes more sense: insert and update rely on the shell (or CRT library) to expand the command line, whereas commands that deal with file names inside the archive such as "delete" and "list" use internal globbing because the CRT/shell can't see those names.

I think this follows the principle of least surprise.

(6) By anonymous on 2021-11-01 18:08:32 in reply to 4 [link] [source]

Documentation gap: the stand-alone sqlar README doesn't mention any deletion capability. You have to read the source to discover the existence of the -d option.

(7) By Warren Young (wyoung) on 2021-11-01 18:12:42 in reply to 6 [link] [source]

You also get it from a bare sqlar command or anything else that doesn't pass the command line argument parser such as sqlar -help. I only chased it down to the C level to show that the feature wasn't added since this thread was started.

Contrast Larry's contribution....

(10) By Kevin Charles (KevinCharles) on 2021-11-01 21:51:44 in reply to 7 [link] [source]

For my monthly backups, I have a delete line, in case I accidentally backup cleartext passwords.

sqlar -d akay_20211001.sqlar akay/pwsafe.txt

Thanks to this thread, I will add a "vacuum" step.

sqlite3 akay_20211001.sqlar "vacuum"

(13) By ddevienne on 2021-11-02 08:02:55 in reply to 10 [link] [source]