[9fans] venti and "contrib": RFC

[t***@polynum.com]

Hello,

Summary of the previous epidodes: My Plan9 installation was still the
initial one as far as partitionning is concerned. Since I had not
grasped the venti purpose, "other" was empty, everything going into
the venti archived. And I was doing a number of install/de-install
of kerTeX for tests purposes, boom!: "disk full" and need to find a way
to load an alternate root to fix things---or reinstall.

But this leads to questions regarding the "contrib" stuff.

When one has the sources, archiving with history the sources make sense.
To take the example of kerTeX, there is a map describing where to put
eventually a file, so the sources vary a little, but the result may be
arbitrary. Secondly, the binaries compiled from the sources may vary
even if the sources do not vary.

So the compiled result is not worth archiving. (The convenience to have
a fallback snapshot to not disrupt work is here; in case of bigger
disaster, the time needed to recompile everything is acceptable---for
kerTeX, even if the result is several tens of Mb, this is a matter of
minutes.) Furthermore, for an "experimental" work, archiving a transient
state is not worth the disk space.

With the design of namespace manipulations, a Plan9 user can "redirect"
the writes where he wants them to happen---venti or not venti, that is
the question.

But the user has to know. Is there a policy described somewhere?

The problem, I think, is that on "other" systems, one thinks backup and
archiving _after_---and decide what goes in backups. While here,
powerful tools are there, by default, but user may be unaware of
consequences. Perhaps should it be proposed by default, for the "let's
see what is Plan9", to get fossil only, and to switch to venti when
things are clear?

Cheers,

[erik quanstrom]

Post by t***@polynum.com
So the compiled result is not worth archiving.

it has been more than once that in tracking down a problem, i've
found that the "known working" executable worked but the source
from that point in history didn't. and vice versa. having the executables
and libraries archived was very valuable.

otoh, just to use round numbers, if your build creates 100mb of new data
and all of it hits venti before being replaced, then you've got 10000
builds/TB. in practice, i think most people push to venti only once
a day, so this is practically infinite. or, put another way, that's
$100/10000 = 1¢/build. since the standard depricated comment is
worth 2¢, it appears that these days 100mb is not worth commenting
on. ☺.

- erik

[t***@polynum.com]

And finally, didn't the increase in size of the disks, with
no decrease

^^^^^^^^^^^^

no increase, of course. If probability of a failure for a sector is P,
increasing the number of sectors increases the probability of disk
failure.

[erik quanstrom]

Post by t***@polynum.com

And finally, didn't the increase in size of the disks, with
no decrease

^^^^^^^^^^^^
no increase, of course. If probability of a failure for a sector is P,
increasing the number of sectors increases the probability of disk
failure.

sector failure != disk failure. disk failure is generally due to the
heads &c, and generally independent of the size of the device.

- erik

[t***@polynum.com]

Post by erik quanstrom

Post by t***@polynum.com
So the compiled result is not worth archiving.

it has been more than once that in tracking down a problem, i've
found that the "known working" executable worked but the source
from that point in history didn't. and vice versa. having the executables
and libraries archived was very valuable.
otoh, just to use round numbers, if your build creates 100mb of new data
and all of it hits venti before being replaced, then you've got 10000
builds/TB. in practice, i think most people push to venti only once
a day, so this is practically infinite. or, put another way, that's
$100/10000 = 1¢/build. since the standard depricated comment is
worth 2¢, it appears that these days 100mb is not worth commenting
on. ?.

Perhaps, but it seems to me like digging ore, extracting the small
percentage of valuable; forging a ring; and throwing it in the ore, and
storing the whole...

Secondly, I still use optical definitive storage from time to time
(disks go in a vault)... with KerGIS and others, and kerTeX, this still
fit 3 times on a CDROM. So...

And finally, didn't the increase in size of the disks, with no decrease
of the reliability, increases the probability of disks failure?
Unfortunately, one finds not "small" (that were huge some years ago)
disks anymore...

PS: and for a Plan9 tester, he will begin by devoting a partition on a
disk to see. The iso is around 300Mb, so allocating 512 or 1024Mb will
seem enough. If he's hooked to Plan9---that happened to me ;)---sooner
or later a problem will occur.

[erik quanstrom]

Post by t***@polynum.com
Perhaps, but it seems to me like digging ore, extracting the small
percentage of valuable; forging a ring; and throwing it in the ore, and
storing the whole...

generally it's apparent which files are worth investigating, and between
history (list of changes by date) and a binary search, it shouldn't take
more than a handful of tries to narrow things down.

in practice, i've found the executables more helpful than the source.

Post by t***@polynum.com
Secondly, I still use optical definitive storage from time to time
(disks go in a vault)... with KerGIS and others, and kerTeX, this still
fit 3 times on a CDROM. So...

if you are using venti, there is no reason to re-archive closed arenas.
(and there's no a priori reason that your optical backup must include
history.)

Post by t***@polynum.com
And finally, didn't the increase in size of the disks, with no decrease
of the reliability, increases the probability of disks failure?
Unfortunately, one finds not "small" (that were huge some years ago)
disks anymore...

i think disk reliability is a term that gets canceled out. if you have
n copies of an executable, whatever the reliablity of the drive, each
copy is exactly as likely to be intact.

Post by t***@polynum.com
PS: and for a Plan9 tester, he will begin by devoting a partition on a
disk to see. The iso is around 300Mb, so allocating 512 or 1024Mb will
seem enough. If he's hooked to Plan9---that happened to me ;)---sooner
or later a problem will occur.

that's not what i did. i started with several 18gb scsi drives.

- erik

[t***@polynum.com]

Post by erik quanstrom

Post by t***@polynum.com
Secondly, I still use optical definitive storage from time to time
(disks go in a vault)... with KerGIS and others, and kerTeX, this still
fit 3 times on a CDROM. So...

if you are using venti, there is no reason to re-archive closed arenas.
(and there's no a priori reason that your optical backup must include
history.)

Because I use CVS (not on Plan9), and I backup my CVS. So, sources with
history. I do not consider CDROM to be eternal. So there is a small
number kept, and the older is destroyed when the new one is burnt.

[erik quanstrom]

Post by t***@polynum.com
Because I use CVS (not on Plan9), and I backup my CVS. So, sources with
history. I do not consider CDROM to be eternal. So there is a small
number kept, and the older is destroyed when the new one is burnt.

sorry. i thought we were talking about organizing plan 9
storage. never mind ....

- erik

[t***@polynum.com]

Post by erik quanstrom

Post by t***@polynum.com
Because I use CVS (not on Plan9), and I backup my CVS. So, sources with
history. I do not consider CDROM to be eternal. So there is a small
number kept, and the older is destroyed when the new one is burnt.

sorry. i thought we were talking about organizing plan 9
storage. never mind ....

I use CVS on NetBSD now. But even on Plan9, I want my sources with
history. This means that on Plan9, I will make a separate partition for
my sources; add a log file to register comments about changes; and
backup the whole arena.

[David du Colombier]

I am not sure to understand your question.

Nothing forces you to dump the full Fossil tree to Venti
every night. You can run snap manually every time you
want, or run it only on a part of the tree.

You can also individually exclude some files from
the snapshots using the DMTMP bit.

If you really want to avoid archiving binaries, you
could simply add a cron job which automatically apply
the DMTMP bit on the binaries just before the
archival snapshot.

Fossil and Venti are very flexible, you can do almost
everything you want.

[t***@polynum.com]

Post by David du Colombier
Fossil and Venti are very flexible, you can do almost
everything you want.

No doubt about that.

But perhaps the other users are smart enough to have understood all this
at installation time, but when I first installed Plan9, that was not for
the archival features. And I spent my time on Plan9 looking for the
distributed system, the namespace and so on, not on venti.

The question is more about the defaults and/or the documentation.

[Russ Cox]

Post by t***@polynum.com
But perhaps the other users are smart enough to have understood all this
at installation time, but when I first installed Plan9, that was not for
the archival features. And I spent my time on Plan9 looking for the
distributed system, the namespace and so on, not on venti.
The question is more about the defaults and/or the documentation.

The default is that you have so little data in comparison to a
modern disk that there is no good reason not to save full
snapshots. As Erik and others have pointed out, if you do
find reason to exclude certain trees from the snapshots, you
can use chmod +t. The system is working as intended.

Russ

[t***@polynum.com]

Post by Russ Cox
The default is that you have so little data in comparison to a
modern disk that there is no good reason not to save full
snapshots. As Erik and others have pointed out, if you do
find reason to exclude certain trees from the snapshots, you
can use chmod +t. The system is working as intended.

Quoting ``Installing the Plan9 Distribution'':

You need an x86-based PC with 32MB of RAM, a supported video card, and a
hard disk with at least 300MB of unpartitionned space and a free primary
partition slot.

Yes, this is from the printed edition of Plan9 Programmer's Manual, 3rd
Edition.

But I don't see why "caveats" will hurt a new comer, who is probably
not devoting an entire new disk to a system he doesn't know yet and
wants to try, but making Plan9 some place on a disk populated with other
data.

And giving a hint about the archival features will not hurt either.

[David du Colombier]

The third edition was published in june 2000. It predates
both Venti (april 2002) and Fossil (january 2003).

This documentation was about installing Plan 9 on a
standalone terminal running kfs, not a file server.

[ron minnich]

I doubt anyone would object if you want to change the text and submit
to the website owners.

ron

[Bakul Shah]

Post by t***@polynum.com

Post by Russ Cox
The default is that you have so little data in comparison to a
modern disk that there is no good reason not to save full
snapshots. As Erik and others have pointed out, if you do
find reason to exclude certain trees from the snapshots, you
can use chmod +t. The system is working as intended.

You need an x86-based PC with 32MB of RAM, a supported video card, and a
hard disk with at least 300MB of unpartitionned space and a free primary
partition slot.
Yes, this is from the printed edition of Plan9 Programmer's Manual, 3rd
Edition.
But I don't see why "caveats" will hurt a new comer, who is probably
not devoting an entire new disk to a system he doesn't know yet and
wants to try, but making Plan9 some place on a disk populated with other
data.

Are you going to update the wiki then? Newbies need all the
help they can get! If you do, make sure to mention they use
RAID1! Not when they are just playing with plan9 but before
they start storing precious data. On a moderen consumer disk,
if you read 1TB, you have 8% chance of a silent bad read
sector. More important to worry about that in today's world
than optimizing disk space use.

If you partition disk space, chances are, it will be used
suboptimally (that is, it will turn out you guessed partition
size wrong for the actual use). These days (for most people)
the *bulk* of their disks contain user data so there is really
no point in partitioning. Just make sure your truly critical
data is backed up separately (repeat until you are satisfied).

[erik quanstrom]

Post by Bakul Shah
if you read 1TB, you have 8% chance of a silent bad read
sector. More important to worry about that in today's world
than optimizing disk space use.

do you have a citation for this? i know if you work out the
numbers from the BER, this is about what you get, but in
practice i do not see this 8%. we do pattern writes all the
time, and i can't recall the last time i saw a "silent" read error.

- erik

[Bakul Shah]

Post by erik quanstrom

Post by Bakul Shah
if you read 1TB, you have 8% chance of a silent bad read
sector. More important to worry about that in today's world
than optimizing disk space use.

do you have a citation for this? i know if you work out the
numbers from the BER, this is about what you get, but in
practice i do not see this 8%. we do pattern writes all the
time, and i can't recall the last time i saw a "silent" read error.

Silent == unseen! Do you log RAID errors? Only way to catch them.

That number is derived purely on an bit error rate (I think
vendors base that on the Reed-Solomon code used). No idea how
"uniformly random" the data (or medium) is in practice. I
thought the "practice" was worse!

[erik quanstrom]

Post by Bakul Shah

Post by erik quanstrom

Post by Bakul Shah
if you read 1TB, you have 8% chance of a silent bad read
sector. More important to worry about that in today's world
than optimizing disk space use.

do you have a citation for this? i know if you work out the
numbers from the BER, this is about what you get, but in
practice i do not see this 8%. we do pattern writes all the
time, and i can't recall the last time i saw a "silent" read error.

Silent == unseen! Do you log RAID errors? Only way to catch them.
That number is derived purely on an bit error rate (I think
vendors base that on the Reed-Solomon code used). No idea how
"uniformly random" the data (or medium) is in practice. I
thought the "practice" was worse!

i thought your definition of silent was not caught by the on-drive
ecc. i think this is not very likely, and we're explicitly checking for
this byrunning massive numbers of disks through pattern writes with
verification, and don't see it.

- erik

[Bakul Shah]

Post by erik quanstrom

Post by Bakul Shah

Post by erik quanstrom

Post by Bakul Shah
if you read 1TB, you have 8% chance of a silent bad read
sector. More important to worry about that in today's world
than optimizing disk space use.

do you have a citation for this? i know if you work out the
numbers from the BER, this is about what you get, but in
practice i do not see this 8%. we do pattern writes all the
time, and i can't recall the last time i saw a "silent" read error.

Silent == unseen! Do you log RAID errors? Only way to catch them.
That number is derived purely on an bit error rate (I think
vendors base that on the Reed-Solomon code used). No idea how
"uniformly random" the data (or medium) is in practice. I
thought the "practice" was worse!

i thought your definition of silent was not caught by the on-drive
ecc. i think this is not very likely, and we're explicitly checking for

Hmm.... You are right! I meant *uncorrectable* read errors
(URE), which are not necessarily *undetectable* errors (where
a data pattern switches to another pattern mapping to the same
syndrome bits). Clearly my memory by now has had much more
massive bit-errors! Still, consumer disk URE rate of 10^-14
coupled with large disk sizes does mean RAID is essential.

Post by erik quanstrom
this byrunning massive numbers of disks through pattern writes with
verification, and don't see it.

Are these new disks? The rate goes up with age. Do SMART
stats show any new errors? It is also possible vendors are
*conservatively* specifying 10^-14 (though I no longer know
how they arrive at the URE number!). Can you share what you
did discover? [offline, if you don't want to broadcast]

You've probably read
http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.ps

[Aram Hăvărneanu]

do you have a citation for this?  i know if you work out the
numbers from the BER, this is about what you get, but in
practice i do not see this 8%.  we do pattern writes all the
time, and i can't recall the last time i saw a "silent" read error.

Yes, the real numbers are much, much lower, but still significant
because they affect RAID reconstruction. See this[1] paper.

Some unrelated, but interesting fact from that paper: "nearline disks
(and their adapters) develop checksum mismatches an order of magnitude
more often than enterprise class disk drives".

[1] L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
Arpaci-Dusseau, and R. H. Arpaci-Dusseau. An Analysis of Data
Corruption in the Storage Stack. In FAST, 2008
http://www.cs.toronto.edu/~bianca/papers/fast08.pdf

[John Floren]

Post by Russ Cox

Post by t***@polynum.com
But perhaps the other users are smart enough to have understood all this
at installation time, but when I first installed Plan9, that was not for
the archival features. And I spent my time on Plan9 looking for the
distributed system, the namespace and so on, not on venti.
The question is more about the defaults and/or the documentation.

The default is that you have so little data in comparison to a
modern disk that there is no good reason not to save full
snapshots. As Erik and others have pointed out, if you do
find reason to exclude certain trees from the snapshots, you
can use chmod +t. The system is working as intended.
Russ

For reference, I set up our current Plan 9 system about half a year
ago. We have 3.8 TB of Venti storage total. We have used 2.8 GB of
that, with basically no precautions taken to set anything +t; in
general, if it's around at 4 a.m., it's going into Venti. I figure we
have roughly another 2,000 years of storage left at the current rate
:)



John

[John Floren]

Post by John Floren
For reference, I set up our current Plan 9 system about half a year
ago. We have 3.8 TB of Venti storage total. We have used 2.8 GB of
that, with basically no precautions taken to set anything +t; in
general, if it's around at 4 a.m., it's going into Venti. I figure we
have roughly another 2,000 years of storage left at the current rate
:)

I first read that "2.8 GB" as 2.8 TB and was utterly confused!
You'd save a bunch of energy if you only powered up venti
look at /n/dump). Though venti might have fits! And the disks
might too! So may be this calls for a two level venti? First
to an SSD RAID and a much less frequent venti/copy to hard
disks.
venti doesn't have a "scrub" command, does it? zfs scrub was
instrumental in warning me that I needed new disks.

Well, we need the venti disks powered on whenever we're using it,
right? Since most of the filesystem is actually living on Venti and
fossil just has pointers to it? Also, I think it's probably better
for disks to stay on all the time rather than go on-off-on-off.

And compared to the rest of the machine room, keeping a Coraid running
all the time isn't that big of a thing.



John

[ron minnich]

but john, the whole your venti would easily fit in even a small server
memory, now and forever ;)

ron

[Bakul Shah]

Post by John Floren
For reference, I set up our current Plan 9 system about half a year
ago. We have 3.8 TB of Venti storage total. We have used 2.8 GB of
that, with basically no precautions taken to set anything +t; in
general, if it's around at 4 a.m., it's going into Venti. I figure we
have roughly another 2,000 years of storage left at the current rate
:)

I first read that "2.8 GB" as 2.8 TB and was utterly confused!

You'd save a bunch of energy if you only powered up venti
disks once @ 4AM for a few minutes (and on demand when you
look at /n/dump). Though venti might have fits! And the disks
might too! So may be this calls for a two level venti? First
to an SSD RAID and a much less frequent venti/copy to hard
disks.

venti doesn't have a "scrub" command, does it? zfs scrub was
instrumental in warning me that I needed new disks.

[erik quanstrom]

You'd save a bunch of energy if you only powered up venti
look at /n/dump). Though venti might have fits! And the disks
might too! So may be this calls for a two level venti? First
to an SSD RAID and a much less frequent venti/copy to hard
disks.
venti doesn't have a "scrub" command, does it? zfs scrub was
instrumental in warning me that I needed new disks.

they're using coraid storage. all this is taken care of for them
by the SR appliance.

- erik

[Aram Hăvărneanu]

venti doesn't have a "scrub" command, does it? zfs scrub was
instrumental in warning me that I needed new disks.

they're using coraid storage.  all this is taken care of for them
by the SR appliance.

Out of curiosity, how? ZFS blocks are checksummed. ZFS scrub reads
not physical blocks on disks, but logical ZFS blocks and validates
their checksum. How can the Coraid appliance determine if the data it
reads is valid or not since it works below the file system layer and
only understands physical blocks?

[erik quanstrom]

Post by Aram Hăvărneanu

venti doesn't have a "scrub" command, does it? zfs scrub was
instrumental in warning me that I needed new disks.

they're using coraid storage.  all this is taken care of for them
by the SR appliance.

Out of curiosity, how? ZFS blocks are checksummed. ZFS scrub reads
not physical blocks on disks, but logical ZFS blocks and validates
their checksum. How can the Coraid appliance determine if the data it
reads is valid or not since it works below the file system layer and
only understands physical blocks?

all redundant raid types have parity (even raid 1; fun fact!).

- erik

[Bakul Shah]

Post by erik quanstrom

You'd save a bunch of energy if you only powered up venti
look at /n/dump). Though venti might have fits! And the disks
might too! So may be this calls for a two level venti? First
to an SSD RAID and a much less frequent venti/copy to hard
disks.
venti doesn't have a "scrub" command, does it? zfs scrub was
instrumental in warning me that I needed new disks.

they're using coraid storage. all this is taken care of for them
by the SR appliance.

When are you going to sell these retail?!

The question was for venti though.

[erik quanstrom]

Post by Bakul Shah

Post by erik quanstrom

venti doesn't have a "scrub" command, does it? zfs scrub was
instrumental in warning me that I needed new disks.

they're using coraid storage. all this is taken care of for them
by the SR appliance.

When are you going to sell these retail?!
The question was for venti though.

i'm not sure i follow. why can't venti assume a perfect array-of-bytes
device and let the appliance take care of it?

if you care enough to get ecc memory, your data path should be
100% ecc protected.

- erik

[Bakul Shah]

Post by erik quanstrom

Post by Bakul Shah

Post by erik quanstrom

venti doesn't have a "scrub" command, does it? zfs scrub was
instrumental in warning me that I needed new disks.

they're using coraid storage. all this is taken care of for them
by the SR appliance.

When are you going to sell these retail?!
The question was for venti though.

i'm not sure i follow. why can't venti assume a perfect array-of-bytes
device and let the appliance take care of it?

If you scrub only the live data, total scrub time becomes a
function of used space as opposed to total space. But the
main reason for asking was that I was thinking of situations
where your appliance is not used but venti is.

The illusion of a perfect array-of-bytes abstraction can't
always be sustained in any case.

Thanks to everyone who set me straight on fossil + venti.

[Steve Simon]

You'd save a bunch of energy if you only powered up venti
look at /n/dump).

If fossil is setup to dump to venti then it needs venti to
work at all. Fossil is a write cache, so, just after the dump
at 4am fossil is empty and consists only of a pointer to the
root of the dump in venti; all reads are then satisfied from
venti alone, until some data is written.

-Steve

[Yaroslav]

You'd save a bunch of energy if you only powered up venti
look at /n/dump).  Though venti might have fits! And the disks
might too! So may be this calls for a two level venti? First
to an SSD RAID and a much less frequent venti/copy to hard
disks.

I think you're confusing kenfs+worm with fossil+venti in sense that
ken fs is a complete cache for worm while fossil is a write cache for
venti. You need venti running all the time.

[erik quanstrom]

Post by Yaroslav

You'd save a bunch of energy if you only powered up venti
look at /n/dump).  Though venti might have fits! And the disks
might too! So may be this calls for a two level venti? First
to an SSD RAID and a much less frequent venti/copy to hard
disks.

I think you're confusing kenfs+worm with fossil+venti in sense that
ken fs is a complete cache for worm while fossil is a write cache for
venti. You need venti running all the time.

this only could work if you assume that everything in the worm
is also in the cache, and you've configured your cache to cache the worm.
since these days the cache is often similar in performance to the worm, the
default we use is to not copy the worm into the cache. this would just
result in more i/o.

so tl;dr you need the worm available at all times be it venti+fossil
or ken fs/cwfs

- erik

[erik quanstrom]

Post by John Floren
For reference, I set up our current Plan 9 system about half a year
ago. We have 3.8 TB of Venti storage total. We have used 2.8 GB of
that, with basically no precautions taken to set anything +t; in
general, if it's around at 4 a.m., it's going into Venti. I figure we
have roughly another 2,000 years of storage left at the current rate
:)

in 10 years, we've managed to store only 645gb of stuff in ken fs. so duplicates
are stored as dupes. large chunks of it are email (from the pre nupas days)
or imported from even older systems. the worm is only 3tb.

the system was built last nov with the then-the-best-option 1tb drives. today,
the whole worm could be put on 2 drives with raid 10. in 2 years, when it's
time to replace the drives, the worm should be less than 2/3 full, assuming
<~300% year/year acceleration in storage used.

my personal system, a mere 6 years old, only has about 12.5gb of junk.
(in a 1500gb worm!). and drives are ripe for replacement.

by the way, in thinking a bit more about the BER and scrubbing, my
3 raids have been scrubbing continuously for 3 years and, (so that's
hundreds of times) except when i actually had a bad disk, i have not
seen URE.

- erik

[t***@polynum.com]

Post by ron minnich
I doubt anyone would object if you want to change the text and submit
to the website owners.

That was my intention, but before, I wanted to submit to the list some
stuff, in order to not publish nonsense. [But probably some people equal
Laronde and nonsense...]

I do want to submit the stuff for revue, since I have been playing with
the installation process to use it at a reparation tool.

Mounting the CD image, extracting the El Torito 2.88Mb image;
mounting the file as a fat; extracting 9load and repopulating my
plan9:9fat (9pcflop having a kernel and a spartiate root is then
a reparation tool). Jivaro'ing the CD image to put it in the 100
Mb 9fat, in order to install from there, Plan9 fat being seen from
NetBSD by playing with a disklabel. Etc.

I don't care about a software incident if I have fun and use it to
learn...

[t***@polynum.com]

Post by John Floren
For reference, I set up our current Plan 9 system about half a year
ago. We have 3.8 TB of Venti storage total. We have used 2.8 GB of
that, with basically no precautions taken to set anything +t; in
general, if it's around at 4 a.m., it's going into Venti. I figure we
have roughly another 2,000 years of storage left at the current rate
:)

The TB were there because you planned to use TeXlive, that's all... ;)