[9fans] usbether

[Richard Miller]

I've been struggling to get the raspberry pi's built-in usb ethernet
adapter working. I can send and receive packets reliably enough to
get a remote file system mounted, but with any kind of heavy use the
usbether input seems to be missing a lot of packets, and everything
gets horribly slow.

After running out of 9pi-specific things to debug, it occurred to me
to try a usb ethernet dongle on an x86 plan 9 machine. There I
observed the same thing: so many dropped packets that the connection
is unusable.

So, has anyone had success using usbether to connect a plan 9 system
to the outside world? I am hoping someone can give me an encouraging
report. I'm a bit worried that it's a fundamental problem with the
plan 9 usb architecture, which is basically synchronous - the host
adapter driver in the kernel will poll a device for input only when
the user-level driver process does a read. This is ok for things like
usbdisk which have an rpc-like protocol, but seems less well suited to
things like ethernet and serial interfaces, where the equivalent
non-usb kernel drivers use qio to read ahead into a queue of buffers
until the user-level consuming process gets around to reading them.

Would anyone like to share experiences or comments?

[Gorka Guardiola]

Post by Richard Miller
So, has anyone had success using usbether to connect a plan 9 system
to the outside world? I am hoping someone can give me an encouraging
report. I'm a bit worried that it's a fundamental problem with the
plan 9 usb architecture, which is basically synchronous - the host
adapter driver in the kernel will poll a device for input only when
the user-level driver process does a read. This is ok for things like
usbdisk which have an rpc-like protocol, but seems less well suited to
things like ethernet and serial interfaces, where the equivalent
non-usb kernel drivers use qio to read ahead into a queue of buffers
until the user-level consuming process gets around to reading them.
Would anyone like to share experiences or comments?

I had a similar problem with serial a while ago. The problem was that
it was taking too long doing too many system calls, doing big reads
and buffering may help you?

G.

[erik quanstrom]

Post by Gorka Guardiola
I had a similar problem with serial a while ago. The problem was that
it was taking too long doing too many system calls, doing big reads
and buffering may help you?

why doesn't the device do the buffering? or is the problem
that usb/ether essentially an event loop of <read packet> <write packet>
and any delays in that pipeline accumulate?

- erik

[Richard Miller]

Post by erik quanstrom
why doesn't the device do the buffering?

By "device" do you mean the hardware usb/ether adapter? It does some
buffering - you can configure it to "burst" as many ether input packets
as will fit in N 512-byte usb packets, in each usb input operation.
I've got N=37 because that's what the linux driver does; of course
there's no documentation to explain what the allowed range is or
what the consequences of varying it would be. As far as I can see
there's no "overrun" flag for the device to tell me I've missed
reading some packets.

Post by erik quanstrom
or is the problem
that usb/ether essentially an event loop of <read packet> <write packet>
and any delays in that pipeline accumulate?

Almost: it's a pipeline of one thread reading a buffer full of packets,
splitting it up, and sending a packet at a time to a second thread, which
writes them to the kernel packet ethernet interface, which stores them
in a Buf queue.

[Gorka Guardiola]

Post by Richard Miller
Almost: it's a pipeline of one thread reading a buffer full of packets,
splitting it up, and sending a packet at a time to a second thread, which
writes them to the kernel packet ethernet interface, which stores them
in a Buf queue.

Taking a cursory glance it looks as if it just reading at most one
full packet at a time
(the size of the buffer is 2000), it may be if they are small you may
get more than one
in one read, but it may not be so, depends on the device and what is
set as maxpkt
which may be changed with usbctl if the device lets you

Making a bigger buffer (after making sure you can actually read more
bytes) and or decoupling
reader and processer by having a buffered channel so that the
reader can spend more time reading may help (it helped for serial).


G.

[Richard Miller]

Post by Gorka Guardiola
Taking a cursory glance it looks as if it just reading at most one
full packet at a time
(the size of the buffer is 2000)

That's the asix driver: the raspberry pi adapter has an SMSC LAN95xx,
see /n/sources/contrib/miller/usb/ether/smsc.c for my driver which
reads up to 37*512=18944 bytes at a time.

[Gorka Guardiola]

On 29/09/2012, at 23:25, Richard Miller <***@hamnavoe.com> wrote:


That's the asix driver: the raspberry pi adapter has an SMSC LAN95xx,
see /n/sources/contrib/miller/usb/ether/smsc.c for my driver which
reads up to 37*512=18944 bytes at a time.


And is it really doing it? I mean it asks for many bytes but how many
does it get?

G.

[Richard Miller]

Post by Gorka Guardiola
And is it really doing it? I mean it asks for many bytes but how many
does it get?

It often gets about 8k at a time (a complete 9p message). I'll have
to add a check to see what the maximum is.

[erik quanstrom]

Post by Richard Miller

Post by Gorka Guardiola
And is it really doing it? I mean it asks for many bytes but how many
does it get?

It often gets about 8k at a time (a complete 9p message). I'll have
to add a check to see what the maximum is.

of course the 64bit question is, which queue is dropping packets.
there are so many to choose from.

- erik

[Richard Miller]

Post by erik quanstrom
of course the 64bit question is, which queue is dropping packets.
there are so many to choose from.

Looks my initial guess was wrong. I added some readahead to the kernel
usb driver and that didn't help at all. Then I reduced the device's
buffering ("burst factor"), and increased the intermediate buffering
between input and output processes in the generic part of the usb/ether
driver. That seems to have done the trick: no more dropped packets.

That's with the raspberry pi built-in usb ether. My asix-based ethernet
dongle doesn't work any better on an x86 host; but at least it doesn't
work any worse. Geoff has pushed the new usb/ether code onto sources.
Anybody who is actually using usb/ether - could you try the new version
and let us know if we've broken yours?

[Charles Forsyth]

the qopen sets the queue limit to 8k, which seems a little low, unless it's
reset later.
the later unchecked qpass (with comment) will toss the data away, which is
a little drastic.

Post by Richard Miller
It often gets about 8k at a time (a complete 9p message). I'll have
to add a check to see what the maximum is.

[Richard Miller]

Post by Charles Forsyth
the qopen sets the queue limit to 8k

Which qopen are you referring to?

[Charles Forsyth]

the wrong one. never mind.

Post by Richard Miller
Which qopen are you referring to?

[erik quanstrom]

Post by Richard Miller

Post by erik quanstrom
why doesn't the device do the buffering?

By "device" do you mean the hardware usb/ether adapter? It does some

i ment #u itself. sorry for being unclear. actually, echoing
gorka, i think there's going to need to be enough threads/buffering
to decouple the packet pipeline.

it might be that the usb packet shuffler should be in the kernel, and
usbd should poke it from user land. for a device without more
processing power than a cray 2, this could make a big difference.
i wrote a sd loopback device that turns a file into a sd device;
perhaps the same model would work for ether? (sdloop is in 9atom.)

- erik

[Charles Forsyth]

the host adapter driver in the kernel will poll a device for input only

when

the user-level driver process does a read

I'm fairly sure the original driver could post a batch of polling requests,
for just that reason,
but it was a long time ago.

[Tristan]

Post by Richard Miller
After running out of 9pi-specific things to debug, it occurred to me
to try a usb ethernet dongle on an x86 plan 9 machine. There I
observed the same thing: so many dropped packets that the connection
is unusable.
So, has anyone had success using usbether to connect a plan 9 system
to the outside world? I am hoping someone can give me an encouraging
report.

Not usb/ether, but my usb/wifi (which is very much based on it) written
for the Libertas (in OLPC) wireless worked a lot better than the any of
my usb ethernet dongles. I never have gotten around to figuring out why.

Good luck!
Tristan