[9fans] venti backup

[hugo rivera]

Hello,
I've backed up all my *active* arenas in to another disk, just to be
safe. The man page says that the index and the bloom filter may be
rebuilt if lost. So it seems sufficient to backup my arenas, am I
right?
saludos,

[Steve Simon]

Post by hugo rivera
So it seems sufficient to backup my arenas, am I
right?

Yes, exactly, I haev done this several times.

It might take a few hours and some studying of manuals
but the arenas are all you need.

-Steve

[hugo rivera]

Great. Thanks.
I hope I will never need to use my backed up arenas :-)

Post by Steve Simon

Post by hugo rivera
So it seems sufficient to backup my arenas, am I
right?

Yes, exactly, I haev done this several times.
It might take a few hours and some studying of manuals
but the arenas are all you need.
-Steve

[David du Colombier]

Post by hugo rivera
I've backed up all my *active* arenas in to another disk, just to be
safe. The man page says that the index and the bloom filter may be
rebuilt if lost. So it seems sufficient to backup my arenas, am I
right?

Yes, you can rebuild the index and the Bloom filter
with 'venti/buildindex -b'.

[hugo rivera]

Just to make sure I could rebuild things in case I should, I've tried
to recover everything from my backed up arenas, but I failed. I am not
sure if things go wrong because the backup per se is wrong or I am
making a mistake while recovering from the backup (or both).
So this is how I create the back up of my active arenas

% venti/checkarenas -v /dev/da1s4
output supresed
arena='arenas059' [31676186624,32213057536)
version=5 created=1326893834 modified=1326893851
clumps=34,617 compressed clumps=17,908 data=274,318,099
compressed data=198,689,879 disk storage=200,870,750
% 9 dd -if /dev/da1s4 -of fa -bs 8192 -count 3932258 #
8192*3932258 = 32213057536

Then I create a zeroed file for the bloom filter and another for the
index section. After I format those files I run

% venti/buildindex -b venti.conf
2012/0119 17:11:41 err 2: invalid ending address in arena table
venti/buildindex: can't init venti: can't initialize venti:
/home/hugo/tmpventi/fa: invalid ending address in arena table

It seems that the backup I create is not correct, am I right?

Post by David du Colombier

Post by hugo rivera
I've backed up all my *active* arenas in to another disk, just to be
safe. The man page says that the index and the bloom filter may be
rebuilt if lost. So it seems sufficient to backup my arenas, am I
right?

Yes, you can rebuild the index and the Bloom filter
with 'venti/buildindex -b'.
--
David du Colombier

[David du Colombier]

Post by hugo rivera
It seems that the backup I create is not correct, am I right?

You truncated your arena partition. The new partition size
doesn't match the size specified in ArenaPart.

You should format a new arena partition, then copy arenas
from the beginning of the first to the end of the last.

[David du Colombier]

To clarify things.

You backup is correct, but it's not necessary to backup the
first 128 blocks of the arena partition. Its only contains
the Venti configuration and the ArenaPart structure.

Here is an example of what I described in my precedent message.

Create an arena partition at least as big as your original
one, and format it:

% dd -if /dev/zero -of arenas.img -bs 8192 -count 4194304
% venti/fmtarenas arenas arenas.img

Then copy your Venti arenas from the beginning of arenas00
(128*8192) to the end of arenas059 (3932258*8192):

% dd -if /dev/da1s4 -of arenas.img -bs 8192 -seek 128 -oseek 128 -count 3932258

Of course, '/dev/da1s4' can be 'fa' in your example.
If you removed the first 128 blocks, you don't have
to use '-seek'.

Finally, write the Venti configuration, rebuild the index
and Bloom filter, and start Venti.

[hugo rivera]

There's something weird going on. First checkarenas reports

% venti/checkarenas -v /dev/da1s4
arena='arenas00' [802816,537673728)
version=5 created=1265030300 modified=1265248834 sealed
score=f383ebf9edefe8d37733c8caba6ff53e8b5517b0
clumps=82,908 compressed clumps=22,812 data=669,897,790
compressed data=531,617,136 disk storage=536,840,340

that's only 98 blocks of 8192 bytes, not 128 as you mention.
Anyway, I run fmtarenas on fa and then if I

% 9 dd -if /dev/da1s4 -of fa -bs 8192 -iseek 98 -oseek 98 -count 65536

or

% # this doesn't make sense but I've tried it, nevertheless
% 9 dd -if /dev/da1s4 -of fa -bs 8192 -iseek 128 -oseek 128 -count 65634

I get

2012/0120 11:57:10 err 2: arena set has wrong magic number: 00000000
expected ArenaPartMagic (0xa9e4a5e7)
venti/buildindex: can't init venti: can't initialize venti: fa: arena
set has wrong magic number: 00000000 expected ArenaPartMagic
(0xa9e4a5e7)

when I run buildindex, and checkarenas says the same thing.

Post by David du Colombier
To clarify things.
You backup is correct, but it's not necessary to backup the
first 128 blocks of the arena partition. Its only contains
the Venti configuration and the ArenaPart structure.
Here is an example of what I described in my precedent message.
Create an arena partition at least as big as your original
% dd -if /dev/zero -of arenas.img -bs 8192 -count 4194304
% venti/fmtarenas arenas arenas.img
Then copy your Venti arenas from the beginning of arenas00
% dd -if /dev/da1s4 -of arenas.img -bs 8192 -seek 128 -oseek 128 -count 3932258
Of course, '/dev/da1s4' can be 'fa' in your example.
If you removed the first 128 blocks, you don't have
to use '-seek'.
Finally, write the Venti configuration, rebuild the index
and Bloom filter, and start Venti.
--
David du Colombier

[David du Colombier]

Post by hugo rivera
that's only 98 blocks of 8192 bytes, not 128 as you mention.

Sorry, I got confused. It's 98 blocks on arena partition and
128 blocks on isect partition.

I just tried. This is what I did.

The goal is to manually recopy the first arena from
the first Venti (arenas1.img) to the second Venti (arenas2.img).

It will work for you as long as you adjust the partition
names, the number of arenas and the sizes.

# first venti

% dd -if /dev/zero -of arenas1.img -bs 8192 -count 131072
% dd -if /dev/zero -of isect1.img -bs 8192 -count 16384
% dd -if /dev/zero -of bloom1.img -bs 8192 -count 8192
% venti/fmtarenas arenas arenas1.img
% venti/fmtisect isect isect1.img
% venti/fmtbloom bloom1.img
% echo 'index main
isect isect1.img
arenas arenas1.img
bloom bloom1.img' | venti/conf -w arenas1.img
% venti/fmtindex arenas1.img
% venti=127.1
% venti/venti -c arenas1.img -m 20 -h tcp!127.1!8000
% vac /sys/src/9/pc
% hget http://127.1:8000/index
index=main version=1 blocksize=8192 tabsize=524288
buckets=16287 div=263706
sect=isect for buckets [0,16287) buckmax=215 arena=arenas0 at index [1048576,537903104)
arena='arenas0' on arenas1.img at [802816,537673728)
version=5 created=1327069595 modified=1327069627
written: clumps=669 compressed clumps=600 data=4,251,763 compressed data=1,174,341 storage=1,216,488
indexed: clumps=0 compressed clumps=0 data=0 compressed data=0 storage=0
% venti/sync
% Kill venti | rc
% bc
802816/8192 = 98
537673728/8192 = 65634
65634-98 = 65536

# second venti

% dd -if /dev/zero -of arenas2.img -bs 8192 -count 65633
% dd -if /dev/zero -of isect2.img -bs 8192 -count 16384
% dd -if /dev/zero -of bloom2.img -bs 8192 -count 8192
% venti/fmtarenas arenas arenas2.img
% venti/fmtisect isect isect2.img
% venti/fmtbloom bloom2.img
% echo 'index main
isect isect2.img
arenas arenas2.img
bloom bloom2.img' | venti/conf -w arenas2.img
% venti/fmtindex arenas2.img
% dd -if arenas2.img -of arenapart -bs 8192 -count 97
% dd -if arenas1.img -of arenas -bs 8192 -iseek 97 -count 65536
% cat arenapart arenas > arenas2.img
% rm -f arenapart arenas
% venti/buildindex -b arenas2.img
0 clumps, 16,287 buckets
2012/0120 14:29:29 read index
venti/buildindex: brand-new index, no work to do
2012/0120 14:29:29 arena arenas0: 669 entries
% venti=127.1
% venti/venti -c arenas2.img -m 20 -h tcp!127.1!8000
term% hget http://127.1:8000/index
index=main version=1 blocksize=8192 tabsize=524288
buckets=16287 div=263706
sect=isect for buckets [0,16287) buckmax=215 arena=arenas0 at index [1048576,537903104)
arena='arenas0' on arenas2.img at [802816,537673728)
version=5 created=1327069595 modified=1327069627
written: clumps=669 compressed clumps=600 data=4,251,763 compressed data=1,174,341 storage=1,216,488
indexed: clumps=0 compressed clumps=0 data=0 compressed data=0
storage=0

Honestly, I think you should just use venti/wrarena to write
backed up arenas to a running Venti. It's much easier.

[hugo rivera]

Thanks a lot, David, for your detailed reply.
I've followed your indications and now I am able to recover from my
venti backup :-)
I must confess that I am puzzled, because some sizes and most seeks
for dd are off by 1 block from what I expect. Particulary,
why do you
% dd -if arenas2.img -of arenapart -bs 8192 -count 97
% dd -if arenas1.img -of arenas -bs 8192 -iseek 97 -count 65536
when the first arena starts after 98 blocks from the beginning of the file?

Post by David du Colombier

Post by hugo rivera
that's only 98 blocks of 8192 bytes, not 128 as you mention.

Sorry, I got confused. It's 98 blocks on arena partition and
128 blocks on isect partition.
I just tried. This is what I did.
The goal is to manually recopy the first arena from
the first Venti (arenas1.img) to the second Venti (arenas2.img).
It will work for you as long as you adjust the partition
names, the number of arenas and the sizes.
# first venti
% dd -if /dev/zero -of arenas1.img -bs 8192 -count 131072
% dd -if /dev/zero -of isect1.img -bs 8192 -count 16384
% dd -if /dev/zero -of bloom1.img -bs 8192 -count 8192
% venti/fmtarenas arenas arenas1.img
% venti/fmtisect isect isect1.img
% venti/fmtbloom bloom1.img
% echo 'index main
   isect isect1.img
   arenas arenas1.img
   bloom bloom1.img' | venti/conf -w arenas1.img
% venti/fmtindex arenas1.img
% venti=127.1
% venti/venti -c arenas1.img -m 20 -h tcp!127.1!8000
% vac /sys/src/9/pc
% hget http://127.1:8000/index
index=main version=1 blocksize=8192 tabsize=524288
   buckets=16287 div=263706
   sect=isect for buckets [0,16287) buckmax=215 arena=arenas0 at index [1048576,537903104)
   arena='arenas0' on arenas1.img at [802816,537673728)
   version=5 created=1327069595 modified=1327069627
   written: clumps=669 compressed clumps=600 data=4,251,763 compressed data=1,174,341 storage=1,216,488
   indexed: clumps=0 compressed clumps=0 data=0 compressed data=0 storage=0
% venti/sync
% Kill venti | rc
% bc
802816/8192 = 98
537673728/8192 = 65634
65634-98 = 65536
# second venti
% dd -if /dev/zero -of arenas2.img -bs 8192 -count 65633
% dd -if /dev/zero -of isect2.img -bs 8192 -count 16384
% dd -if /dev/zero -of bloom2.img -bs 8192 -count 8192
% venti/fmtarenas arenas arenas2.img
% venti/fmtisect isect isect2.img
% venti/fmtbloom bloom2.img
% echo 'index main
   isect isect2.img
   arenas arenas2.img
   bloom bloom2.img' | venti/conf -w arenas2.img
% venti/fmtindex arenas2.img
% dd -if arenas2.img -of arenapart -bs 8192 -count 97
% dd -if arenas1.img -of arenas -bs 8192 -iseek 97 -count 65536
% cat arenapart arenas > arenas2.img
% rm -f arenapart arenas
% venti/buildindex -b arenas2.img
0 clumps, 16,287 buckets
2012/0120 14:29:29 read index
venti/buildindex: brand-new index, no work to do
2012/0120 14:29:29 arena arenas0: 669 entries
% venti=127.1
% venti/venti -c arenas2.img -m 20 -h tcp!127.1!8000
term% hget http://127.1:8000/index
index=main version=1 blocksize=8192 tabsize=524288
   buckets=16287 div=263706
   sect=isect for buckets [0,16287) buckmax=215 arena=arenas0 at index [1048576,537903104)
   arena='arenas0' on arenas2.img at [802816,537673728)
   version=5 created=1327069595 modified=1327069627
   written: clumps=669 compressed clumps=600 data=4,251,763 compressed data=1,174,341 storage=1,216,488
   indexed: clumps=0 compressed clumps=0 data=0 compressed data=0
   storage=0
Honestly, I think you should just use venti/wrarena to write
backed up arenas to a running Venti. It's much easier.
--
David du Colombier

[David du Colombier]

This is because each arena have an header (ArenaHead) and a trailer
(ArenaTrail) we would like to copy. The header, in particular, is located
just one block before the start of the arena.

--
David du Colombier

[hugo rivera]

Does the presence of the trailer imply that I should add an extra
block to the arenas backup?
If my last arena is

arena='arenas059' [31676186624,32213057536)

then I should backup 32213057536+8192 bytes instead of 32213057536?

Post by David du Colombier
This is because each arena have an header (ArenaHead) and a trailer
(ArenaTrail) we would like to copy. The header, in particular, is located
just one block before the start of the arena.
--
David du Colombier

[David du Colombier]

Post by hugo rivera
Does the presence of the trailer imply that I should add an extra
block to the arenas backup?
If my last arena is
arena='arenas059' [31676186624,32213057536)
then I should backup 32213057536+8192 bytes instead of 32213057536?

No, the trailer is located at the end of the arena,
just after clump info.

The second number you see is the start of the next arena.
It is preceded by two blocks:

- the arena header of the next arena,
- the arena trailer of the precedent arena.

32213057536/8192 = 3932258

In you example, arenas060 start at 3932258, so the header of
arenas060 is 3932257 and the trailer of arenas059 is 3932256.

You should really backup from 1048576-8192 (start of arenas00,
including header) to 32213057536-2*8192 (end of arenas059,
including trailer).

Like many Venti tools, checkarenas check each arena header
and trailer and verify they match. So you should be confident.

[hugo rivera]

OK, thanks a lot for your help!

Post by David du Colombier

Post by hugo rivera
Does the presence of the trailer imply that I should add an extra
block to the arenas backup?
If my last arena is
arena='arenas059' [31676186624,32213057536)
then I should backup 32213057536+8192 bytes instead of 32213057536?

No, the trailer is located at the end of the arena,
just after clump info.
The second number you see is the start of the next arena.
 - the arena header of the next arena,
 - the arena trailer of the precedent arena.
32213057536/8192 = 3932258
In you example, arenas060 start at 3932258, so the header of
arenas060 is 3932257 and the trailer of arenas059 is 3932256.
You should really backup from 1048576-8192 (start of arenas00,
including header) to 32213057536-2*8192 (end of arenas059,
including trailer).
Like many Venti tools, checkarenas check each arena header
and trailer and verify they match. So you should be confident.
--
David du Colombier