large archives

[awgcooper]

I want to try and understand the specific risks associated with having an archive size that is 'large' (unless the answers are the same for both, let's assume two sizes for the purpose of this question: 2-5TB and ≥5TB). I get the basic point that the larger the archive, the larger amount of irrecoverable data there is should the archive become severely corrupted or unreadable. However, is there a causal connection between archive size and the probability of it becoming corrupted, i.e. an archive of size X is A% more likely to become corrupted than one of 0.5X, once X exceeds a certain size. I'm using the term 'corrupted' here to mean 'archive-wide corruption' which is, I appreciate, a pretty bad scenario.

In terms of other potential downsides:

• Is there anything about how Borg works that makes is harder to retrievew files, directories or even a full repo as the size of the archive grows beyond a certain point?
• Initial backup time is a downside but once that's done it's just incrementals (which, let's assume, won't be 'too large'). Is there a point at which archive size affects real time dedup calculations during incremental backups? Can they become more 'inefficient' or subject to potential failure?

Anything else specific?

[ThomasWaldmann]

In borg 1.x there is a (very high) design limitation on the max. archive size, borg info shows that. Have an eye on that if you create very large archives and avoid getting close to 100%. As the limit is quite high, this usually is not a problem. In borg2, that limit will be increased by a big factor, so it will be far beyond any imaginable practical usage.

Other than that, there are some timing and memory usage influences:

• an archive is linearly processed, so time = O(archive_size)
• borg check will have time = O(repo_size + f(sum of all archive sizes))
• memory usage = O(chunks count in repo) + O(files count in data set) (see docs for more precise formula)

I maybe would try to avoid creating "monster repos" and prefer multiple smaller repos, using some natural partitioning of your data. Sometimes there are also other reasons for partitioning, e.g. for pruning one wants to have system file archives separate from user file archives.

In borg 1.x there are some reasons for not using 1 repo for multiple borg clients (see FAQ).

[awgcooper]

My Utilization of maximum supported archive size amount when I run borg info always seems to be 0% whether I run it on the whole repo or a particular archive - say the base/first one which is often the largest. The repo I've been running this on is appx 500GB in deduped sized. Exactly how high is the maximum number?

[ThomasWaldmann]

Not sure, maybe we have it in the docs ("internals" or "faq"). It is a very high limit and if it shows 0% for your archive, it is at least 100x higher than what you have.

[awgcooper]

Ok, will check. But basically at these levels a 2-5TB or, even 10TB, archive isn't going to stress the 1.x architecure and, when it comes along, certainly not 2.x?

[ThomasWaldmann]

IIRC, the issue was about the archive metadata stream size, so it rather depends on the files count and the metadata related to these files than the file content size.

But yeah, guess that usually is far away from the borg 1.x limit.

[SpiritInAShell]

@awgcooper I am having similar thoughts about large archives. Having a 10TiB archive (for maximum deduplication) I conclude, that eg. the time for borg check (on my mediocre hardware) is very painful (running borg check --repository-only is so much faster on storagebox@hetzner). Also, doing backups (local disk/off-site) of that repo with rsync is discouraged because it dulicates any error on a binary level.

Therefor splitting in logical sub units is my goal, accepting loss of some deduplication efficiency.

On the other hand, as with a future borg2 a lossless borg transfer and borg export-tar | borg import is possible, it would be possible for me to have such a "megalo-repo" at one location with smaller and easier-to-handle "sub unit clones" at other locations (disk/on-site/off-site). From experiments with borg2-beta10 borg transfer appears to be a quite good experience.

Last thing is only "safe(r)" if errors in one repository cannot be "populated" to another repository with the above mentioned commands. (as borg transfer apparently does not extract files, but copies fully-deduplicated and compressed blocks)

Just there is a thought @ThomasWaldmann, concerning borg transfer:
a) whether there would be any presumable/known risk of also transferring errors
b) if there is any checking done when reading from BORG_OTHER_REPO and writing to BORG_REPO

[awgcooper]

@SpiritInAShell: I agree about borg check times, not to mention if one adds --verify every now and then (of course no criticism of the software is being made; it's simply a time intensive task).

Nice idea about the possibilities with borg2. I've used it quite a lot but not the transfer functionality. I will give it a go at some point. Let's see what TW has to say.

[ThomasWaldmann]

@SpiritInAShell when using borg2 transfer, it decrypts (that includes authentication == checking a strong cryptographic signature) the source chunks and re-encrypts them for the target repository.

That would catch all corruption that happened after the source chunks were authenticated (which is right after compression and encryption, before they were written to the repo).