Discussion:
plan9: how to get irq to mach routing
(too old to reply)
p9 newbie
2013-01-24 10:00:01 UTC
Permalink
Hi,

I would like to retrieve what IRQs are routed to what Machs (or lapics). I dumped IOAPIC's Redirection Table entries, but Destination bits [63:48] are always '0xFF'. Is there way to determine the routing between IRQs and Machs/LAPICs serving them?. One rudimentary way I tried is that, in the interrupt service routine I accessed the 'm' (Mach *) external variable to figure the routing between the IRQ and and the Mach.. Wondering if there is a better way. (for all legacy, msi and msi-x interrupt types)

Thanks
rg
erik quanstrom
2013-01-24 16:22:16 UTC
Permalink
Hi,
I would like to retrieve what IRQs are routed to what Machs (or
lapics). I dumped IOAPIC's Redirection Table entries, but Destination
bits [63:48] are always '0xFF'. Is there way to determine the routing
between IRQs and Machs/LAPICs serving them?. One rudimentary way I
tried is that, in the interrupt service routine I accessed the 'm'
(Mach *) external variable to figure the routing between the IRQ and
and the Mach.. Wondering if there is a better way. (for all legacy,
msi and msi-x interrupt types)
leaving 8259 interrupts to the side, in a system using apics, there are
four potential sources of interrupts for a processor
- processor traps,
- the lapic. 1 lapic attached to each processor
- i/o apics
- device sending an msi/msi-x interrupt.

obviously lapic interrupts (such as the clock timer) and traps
(such as #PF) are handled by the processor that generated them.
in flat or cluster mode, external interrupts are usually routed to
the "lowest priority" processor. in physical mode, a particular
processor is targeted. msi reuses the same logic.

the distribution uses physical addressing. this means that interrupt
ι always targets processor p. so the high bits should
match the processor picked. see
/n/sources/plan9/sys/src/9/pc/mp.c:793
/n/sources/plan9/sys/src/9/pc/mp.c:824
i have not run this code to check, but it doesn't look like it sets
the redirection to broadcast. (iirc, it's illegal to target a non-existent
processor from an i/o apic.)

i was having trouble with apics a few years ago due to the hardware
and thus did the apic implementation described in the man page.
this implementation supports flat, cluster & physical modes:

http://www.quanstro.net/magic/man2html/3/apic

there are a number of files in '#P' that should help you poke
around such as '#P/mpirq' which prints out the vectors as
programmed.

nix uses the same implementation with msi, but only
uses physical mode in round-robin fashion. i need to
merge these. there's no reason for them to be different.

here's an example from a machine with physical-mode
vectors. the i/o apic targets lapic 00 and 04.
minooka; cat '#P/mpvec' | grep -v '10000$'
8 1 0000000000000141
8 4 0400000000000161

here are the processors so we can find them.
(this format keeps changing. don't depend on it. :))

minooka; cat '#P/mpapic'
proc 0 00000000 00000000 00000000 be 0 0xfee00000
proc 1 01000000 01000000 01000000 e 8 0xfee00000
proc 2 02000000 02000000 02000000 e 1 0xfee00000
proc 3 03000000 03000000 03000000 e 9 0xfee00000
proc 4 04000000 04000000 04000000 e 2 0xfee00000
proc 5 05000000 05000000 05000000 e 10 0xfee00000
proc 6 06000000 06000000 06000000 e 3 0xfee00000
proc 7 07000000 07000000 07000000 e 11 0xfee00000
ioapic 8 00000000 00000000 00000000 e 0 0xfec00000
ioapic 9 00000000 00000000 00000000 e 0 0xfec8a000
proc 10 10000000 10000000 10000000 e 4 0xfee00000
proc 11 11000000 11000000 11000000 e 12 0xfee00000
proc 12 12000000 12000000 12000000 e 5 0xfee00000
proc 13 13000000 13000000 13000000 e 13 0xfee00000
proc 14 14000000 14000000 14000000 e 6 0xfee00000
proc 15 15000000 15000000 15000000 e 14 0xfee00000
proc 16 16000000 16000000 16000000 e 7 0xfee00000
proc 17 17000000 17000000 17000000 e 15 0xfee00000

so this is processor 0 and 4. (that's not normally the case. this
machine is pretty easy.)

for msi,

minooka; /mnt/term/386/bin/pci -m | grep 00000000fee
# flags target addr data next ptr
0.31.2 0009 00000000fee05000 00004069 80
1.0.0 0181 00000000fee02000 00004051 50
1.0.1 0181 00000000fee03000 00004059 50
3.0.0 0081 00000000fee01000 00004049 44

bits 16:12 are the target. so we see that we've targeted lapic 1, 3, 2, 5.
(which are fortunately for this example, identity mapped.)

- erik
p9 newbie
2013-01-25 09:54:16 UTC
Permalink
Thanks for the detailed explanation Eriq. I couldn't find a pci binary that supported -m option.
I have
# pci -help
usage: pci [-vb] [vid/did ...]

Thanks
rg
Post by erik quanstrom
Hi,
I would like to retrieve what IRQs are routed to what Machs (or
lapics). I dumped IOAPIC's Redirection Table entries, but Destination
bits [63:48] are always '0xFF'. Is there way to determine the routing
between IRQs and Machs/LAPICs serving them?. One rudimentary way I
tried is that, in the interrupt service routine I accessed the 'm'
(Mach *) external variable to figure the routing between the IRQ and
and the Mach.. Wondering if there is a better way. (for all legacy,
msi and msi-x interrupt types)
leaving 8259 interrupts to the side, in a system using apics, there are
four potential sources of interrupts for a processor
- processor traps,
- the lapic. 1 lapic attached to each processor
- i/o apics
- device sending an msi/msi-x interrupt.
obviously lapic interrupts (such as the clock timer) and traps
(such as #PF) are handled by the processor that generated them.
in flat or cluster mode, external interrupts are usually routed to
the "lowest priority" processor. in physical mode, a particular
processor is targeted. msi reuses the same logic.
the distribution uses physical addressing. this means that interrupt
ι always targets processor p. so the high bits should
match the processor picked. see
/n/sources/plan9/sys/src/9/pc/mp.c:793
/n/sources/plan9/sys/src/9/pc/mp.c:824
i have not run this code to check, but it doesn't look like it sets
the redirection to broadcast. (iirc, it's illegal to target a non-existent
processor from an i/o apic.)
i was having trouble with apics a few years ago due to the hardware
and thus did the apic implementation described in the man page.
http://www.quanstro.net/magic/man2html/3/apic
there are a number of files in '#P' that should help you poke
around such as '#P/mpirq' which prints out the vectors as
programmed.
nix uses the same implementation with msi, but only
uses physical mode in round-robin fashion. i need to
merge these. there's no reason for them to be different.
here's an example from a machine with physical-mode
vectors. the i/o apic targets lapic 00 and 04.
minooka; cat '#P/mpvec' | grep -v '10000$'
8 1 0000000000000141
8 4 0400000000000161
here are the processors so we can find them.
(this format keeps changing. don't depend on it. :))
minooka; cat '#P/mpapic'
proc 0 00000000 00000000 00000000 be 0 0xfee00000
proc 1 01000000 01000000 01000000 e 8 0xfee00000
proc 2 02000000 02000000 02000000 e 1 0xfee00000
proc 3 03000000 03000000 03000000 e 9 0xfee00000
proc 4 04000000 04000000 04000000 e 2 0xfee00000
proc 5 05000000 05000000 05000000 e 10 0xfee00000
proc 6 06000000 06000000 06000000 e 3 0xfee00000
proc 7 07000000 07000000 07000000 e 11 0xfee00000
ioapic 8 00000000 00000000 00000000 e 0 0xfec00000
ioapic 9 00000000 00000000 00000000 e 0 0xfec8a000
proc 10 10000000 10000000 10000000 e 4 0xfee00000
proc 11 11000000 11000000 11000000 e 12 0xfee00000
proc 12 12000000 12000000 12000000 e 5 0xfee00000
proc 13 13000000 13000000 13000000 e 13 0xfee00000
proc 14 14000000 14000000 14000000 e 6 0xfee00000
proc 15 15000000 15000000 15000000 e 14 0xfee00000
proc 16 16000000 16000000 16000000 e 7 0xfee00000
proc 17 17000000 17000000 17000000 e 15 0xfee00000
so this is processor 0 and 4. (that's not normally the case. this
machine is pretty easy.)
for msi,
minooka; /mnt/term/386/bin/pci -m | grep 00000000fee
# flags target addr data next ptr
0.31.2 0009 00000000fee05000 00004069 80
1.0.0 0181 00000000fee02000 00004051 50
1.0.1 0181 00000000fee03000 00004059 50
3.0.0 0081 00000000fee01000 00004049 44
bits 16:12 are the target. so we see that we've targeted lapic 1, 3, 2, 5.
(which are fortunately for this example, identity mapped.)
- erik
erik quanstrom
2013-01-25 15:31:00 UTC
Permalink
Post by p9 newbie
Thanks for the detailed explanation Eriq. I couldn't find a pci binary that supported -m option.
I have
# pci -help
usage: pci [-vb] [vid/did ...]
there aren't any long options in plan 9, so the old way to say
that was "pci -?". otherwise that should be parsed as 4 options,
all of which might be valid.

i put /n/sources/contrib/quanstor/8.pci up on sources. the
man page is http://www.quanstro.net/magic/man2html.8/pci
it requires that the caller be allowed to read $#/pci/$dev^raw.
this seems safe enough.

- erik

Loading...