[9fans] RAID box with plan9 filesystem

[t***@polynum.com]

Hello,

Perhaps somebody will have hints about the following.

I work with ou for small to middle sized enterprises, that can have the
need for a GIS infrastructure.

The data is the crux; but the evolution of data, and the ability to be
able to view the state of data at YYYY-MM-DD hh:mm:ss to compare
projects or the avancement of realization is essential.

As far as my software is concerned, I have solved easily the problem
because... I'm the developper: I know what to save; I save only the
files that allow the value adding software to rebuild everything if
needed; I save them in text format; and I simply use a CVS... Simple.
But works. And this secondary feature is what prospects welcome first.

But what is here with my software would be interesting for other data,
but in binary format.

The best solution for me would be a RAID 1---or some form of backup---;
the reduction of the size of the stored data by sharing the duplicated
blocks; and the history.

As one can rapidly conclude: This means plan9 type WORM.

So are there commercial products available:

1) As a small file server to plug in a network;

2) With at least RAID 1 hardware;

3) With a Plan9 like WORM filesystem;

4) And an easy interface for---mainly---Windows workstations (the only
alien in these networks being my Unix server). This means a CIFS
protocole.

Constraints:

1) Product available in France;

2) I'm neither a hardware nor a filesystem expert, so I don't plan to
provide my own solution or to administrate the beast: if enterprise X
buys to provider Y, they riot directly together in case of problem, I
don't want to be the middle man...

Thanks for any tip!

[Aram Hăvărneanu]

I want to build a Plan9 storage server for home, so I figured this
thread might be a good place to ask. I know nothing about what
hardware to buy.

I want to use venti for permanent storage. For temporary storage that
might or might not be saved to venti, I don't know, I've been thinking
of cwfs. I've heard it's as reliable as kfs, though I never had any
issue with any Plan9 filesystem.

I know that by default kfs uses something like 10MB RAM for the buffer
cache, is this also true for cwfs? If so, can this be increased?
Does it make sense for cwfs to dump to venti? I wish to use venti
because I'll be using it for other things as well, like backing up my
Windows and Mac machines, not only for cwfs dumps. Should I just use
fossil?

I don't know anything about hardware. I've heard Plan9 software RAID
doesn't notify on hardware failure. Should I use a true, hardware
RAID card (one that doesn't require drivers)? What about the box
itself?

Any tips are appreciated, thanks.

[Steve Simon]

You can use kfs or cwfs with venti but the usual combination is
venti with fossil, this is more of a symbiotic arrangmenet than
venti would be with the others. Cwfs contains an archival server
and a short term store like venti and fossil.

mirroring and striping is possible with cwfs's backend, or with the
fs server which can run under any filesystem.

Erik has some reccomended supermicro dual atom motherboards for a server,
I use one but the older model which draws a bit more power for the glue logic.

Personally I have one of these dual atom motherboards, two Enterprise duty
500Gb SATA3 drives, from different manufacturers (with the hope that one will
fail before the other), and I run fossil +_ venti + a cpu + auth server on this
box as my home server. the disks are mirrored partition by partition, This includes
a 200Gb as a jutebox partition with ripped iso images of usefull CDROMs.

-Steve

[Anthony Sorace]

Post by Aram Hăvărneanu
I want to use venti for permanent storage. For temporary storage that
might or might not be saved to venti, I don't know, I've been thinking
of cwfs. I've heard it's as reliable as kfs, though I never had any
issue with any Plan9 filesystem.

Make sure you're not confusing kfs with Ken's file server, sometimes
(confusingly) called "kenfs". kenfs formed the basis for cwfs, and the
code and function is closely related; kfs is essentially unrelated, and is a
much more conventional disk file system.

Ken's file server is extremely solid, and I believe cwfs is comparable
(although I don't have nearly as much experience with it). kfs is much less
so; I've had it become unrecoverable after unexpected shutdown several
times. My experience with fossil has been somewhere between the two,
but it's very easy to recover if you're using it with venti.

Post by Aram Hăvărneanu
Does it make sense for cwfs to dump to venti? I wish to use venti
because I'll be using it for other things as well, like backing up my
Windows and Mac machines, not only for cwfs dumps. Should I just use
fossil?

Using fossil with venti would give you the most seamless integration. You
can dump any file system to venti using vac, though. Using cwfs with venti
would seem a bit redundant, as you'd now have two unrelated systems
providing archival, write-once storage (unless cwfs grew some form of
venti integration while I wasn't watching).

Post by Aram Hăvărneanu
I've heard Plan9 software RAID doesn't notify on hardware failure. Should
I use a true, hardware RAID card (one that doesn't require drivers)?

It's true that fs(3), which provides RAID-like functionality, doesn't do failure
notification in a particularly useful way. As per the man page:
Mirrors are RAID-like but not RAID. There is no fancy
recovery mechanism and no automatic initial copying from a
master drive to its mirror drives.
I use it and have been happy with it, but it's not a complete solution for
genuinely critical data. I don't know much about RAID cards, but the main
thing I'd want to avoid there is proprietary on-disk formats. It might be a bit
much for home use, but if I had a little bit of a budget I'd use Coraid's AoE
stuff as the basis for my storage.

[Aram Hăvărneanu]

Post by Anthony Sorace
Make sure you're not confusing kfs with Ken's file server, sometimes
(confusingly) called "kenfs". [...]
Ken's file server is extremely solid, and I believe cwfs is comparable
(although I don't have nearly as much experience with it). kfs is much less
so; I've had it become unrecoverable after unexpected shutdown several
times.

I know that kfs and Ken's file server are two different things. I
have heard very good things about the reliability of both. You say
kfs is not very reliable, oh well.

Post by Anthony Sorace
My experience with fossil has been somewhere between the two,
but it's very easy to recover if you're using it with venti.

What does this recovery mean? Does fossil need to be repopulated, an
O(n) operation, or does it act merely as a cache that gets cold? (an
O(1) operation).

Post by Anthony Sorace
Using cwfs with venti
would seem a bit redundant, as you'd now have two unrelated systems
providing archival, write-once storage (unless cwfs grew some form of
venti integration while I wasn't watching).

I did not know cwfs implements a WORM, obviously I expected it to use
a WORM but I didn't know it implemented the WORM itself. It makes a
lot of sense considering the ancestry.

I guess I should stick with fossil.

Post by Anthony Sorace
I don't know much about RAID cards, but the main
thing I'd want to avoid there is proprietary on-disk formats.

I've heard some cards merely mirror the data and the array metadata is
stored in the card's nvram, not on disk. I may be mistaken, I am not
a hardware guy. Obviously this only matters for mirroring, any other
form of RAID requires some form of on-disk format.

Post by Anthony Sorace
It might be a bit
much for home use, but if I had a little bit of a budget I'd use Coraid's AoE
stuff as the basis for my storage.

Yeah, it's pretty overkill. I've previously worked at a storage
company as a file system guy and now I have at home a nice array with
ZFS on top. It works great, but I want to scale down. I want less
stuff, not more. And I want to use Plan9, not Solaris.

Thanks,

[Yaroslav]

Post by Anthony Sorace
My experience with fossil has been somewhere between the two,
but it's very easy to recover if you're using it with venti.

What does this recovery mean?  Does fossil need to be repopulated, an
O(n) operation, or does it act merely as a cache that gets cold? (an
O(1) operation).

Fossil is a write cache for venti - "the blocks in the archival
snapshots are moved to Venti". Read requests are proxied for venti.
Recovery means initializing the filesystem using the given vac scrore,
with no implied data transfer - blocks are copied-on-write.

[Aram Hăvărneanu]

Post by Yaroslav
Recovery means initializing the filesystem using the given vac scrore,
with no implied data transfer - blocks are copied-on-write.

Thanks for the explanation.

[Charles Forsyth]

Many file servers using disk drives are "unreliable" if you shut them down
without saving data in memory first.
Both /sys/src/fs and disk/kfs force certain metadata updates to disk first
to try to ensure that the fs structure,
if not the content, remains consistent. In fact, disk/kfs does more than
the old file server (it forces indirect block
updates out as well). Disk/kfs only seemed less reliable than the old file
system because it is more likely
to be shut down without syncing. Also, the old file server could be
recovered from a consistent root
if the worm dump was used. If you sync, you shouldn't have too much trouble.
Of course, forcing write-through makes the file system slower than some
others
for updates. At the cost of some code complexity, fossil attempts to do
better, using a "soft update"
scheme to ensure or attempt to ensure that everything on disk is
consistent. It's mentioned
briefly in the paper.

Post by Aram Hăvărneanu
I
have heard very good things about the reliability of both. You say
kfs is not very reliable, oh well.