HitchHiker Guide to the ENP-2611

So you just have installed your shiny Radisys ENP card in a brand-new computer (or an old computer if you managed to find one with 64-bit PCI) and starts getting disappointed by the lack of a CD with manuals, support,  etc. When first reading the manuals from Radisys, i was scared by the recommendations that "your bus must support 64-bit 66MHz PCI with +3.3V +5.0V and +12V power supply" ... Talking with Lukas Ruf revealed that i was worrying too much, and plugging the card in a DELL Poweredge server proved he was indeed right. Nothing has been gone in smoke so far, the fan is cooling and the XScale is working fine.

Radisys also insists on the fact the host machine should be running RedHat 7.something... I'm now running SuSE 9.x with a 2.6 kernel and everything is fine for me. Just pick up the distribution you know the best and that features network administration tools you feel the most comfortable.

First, get back on radisys website with your serial number. there, you will find a collection a resources (including the must-have "programmer manual") and a form to get access to the file manager where Linux Support Packages (LSP) are available for download. This page is not a substitute to those resources; rather a guide through potential issues you could encounter.

Also contact MontaVista to get a previewkit for one of the IXP2xxx based systems. Radisys no longer provides it. And better ask it soon, because it took me about 2 weeks of mails and phonecalls to get the software on my machine. With patches provided at ixp2xxx.sf.net and busybox package, you might have your card working without MontaVista support, but ...

Getting the thing wired.

Your ENP-2611 card has 3 gigabit ethernet connectors, but these are the last things you'll be using. If you can plug fibers there now, fine. If you can't, that's not big deal. Instead, locate the management port: it's a traditionnal RJ-45 connector (if you haven't optical ports, the gigabit ports are still in the "SFP" format, that is a connector that can be plugged-out and replaced, while the management port is simply soldered on the card. Plug it on the same LAN as the "host" machine.

    note that on some cards, including mine, the management port is inside of the computer.

Using your "null modem" cable (shipped with the card), connect the 3-pin UART slot of your card to the serial port of the host PC. make sure your system has minicom installed -- or any equivalent program you're happy with. I had no knowledge of serial-based management before this so i just stick to minicom.

First connexion to your card

The easiest way to get started is through the null-modem cable. As soon as you powered on the host machine, your ENP card has been waiting for operations (yes, all that long) on that interface. The NULL modem device of your host machine is usually /dev/ttyS0 and the minicom program assume he'll use /dev/modem. If you know a better way than symlink'ing modem to ttyS0, do it ...

/dev# ln -s /dev/ttyS0 modem

will just be fine. start now minicom and setup 57600 bauds, "8N1" communication mode. Make sure you pick the proper terminal settings (VT102 with backspace) and you should be ready.

The factory defaults for the ENP card is to run the diagnostic tool after 2 seconds if the user has not hit CTRL+C by that time, so you can type "help" in the minicom to see what's available there, but with the exception of show, none of them are really useful, so just hit the red-reset-button and press CTRL+C as you're prompted to access the RedBoot manager console.

RedBoot> is your bootloader for your ENP card, much like GRUB is for your host PC (still using Lilo ? man, you must be kidding, right ?). It comes with different functions that lets you probe and test the hardware (yes, running "test all" on RedBoot while reading could be a good idea) and (of course) load and run software.Unless you encounter a situation that really requires it, stay away from RedBoot upgrading, and from any of the flash-manipulation commands: there is no ROM containing RedBoot or any recovering tool so if you trash your flash, you'll have to return the card to the factory.

Take your time to read about the load command, the memory map of the card and to toy with the fis list commands.

Loading something on the card

This is a section where the "programmer guide" from Radisys is extremely detailed and useful. Loading a linux kernel on the ENP is no different from loading a Unix kernel on a diskless X workstation, except that we're now in the 21th century :P ... Chapter 2 of programmer's guide should help you with

• DHCP configuration: your host (or any other machine on the lab LAN) should be running a DHCP daemon for giving the card its own IP address. If you succesfully configure it, RedBoot should report the management port's IP address and the server it has used just after memory scrubbing. You'll notice that it reports MAC address aswell (which you might need in your DHCP configuration). If your card managed to obtain an IP address, you should now be able to ping it, too.
  
• TFTP configuration: this is the protocol you'll use when loading linux kernels. Here too, it's more convenient to have it on the host. So put any file in /tftpboot and issue the load -v -r -b 0x02000000 -m tftp -h xxx.yyy.zzz.uuu /tftpboot/any-file command to check if it can work properly.
  

This last command instructed RedBoot to transfer any-file through the Ethernet 100Mbps LAN and to store it in DRAM at 0x0200,0000 (that is, 32MB), and to keep it "as is". Unlike GRUB, RedBoot doesn't understand zImage of linux kernels, nor even ELF files. the '-r' flag thus tells the file is to be loaded raw, by comparison to e.g. the diag.bin file for diagnostic.

The zImage files are built so that they piggy-back the vmlinux image that they will unpack at the proper "load address". On the ARM architecture, the zImage unpacker is position-independent, so whether you prefer -b 0x02000000 or -b 0x0c028000 for loading is completely up to you. Still, keep in mind that you need to load the image in RAM. With a card having 256MB, any address above 0x1000,0000 will be refused (0x2000,0000 on a card with 512MB, etc.) ... Make sure, too, that the address where you load the zImage file doesn't conflict with the address where the unpacker will put the image. Images provided at ixp2xxx.sourceforge.net are happily loaded at 0x0c02,8000 -- the 'magic' address used by  the docs that i'll use here too.

note: if your host machine runs a firewall, you may experience extra trouble during DHCP or TFTP phases. In case of doubt, try again with all rules desactivated, then enable the rules again.

Running the Linux image

So you have your image loaded. Fine. You can now try to get it working with exec 0x0c028000 command from RedBoot.

note: if you're running your own linux image, or something you compiled out of MontaVista linux sources, make sure you have a load-address suitable for your system and that you defined proper baudrate for the console. invoking strings vmlinux should show you something like console=ttyS0,57600 ip=bootp root=/dev/nfs gdbbaud=57600 gdbttyS=1 nohalt

While the manuals recommands using go 0xc028000, the image built at ixp2xxx.sf.net prefer the use of "exec" which will provide additionnal info about the machine which allow the same image to work with more systems. If "exec" isn't supported by your RedBoot, there are patches that let you build a Linux kernel specifically for your machine. So,

At this stage, your presence near the card is no longer really required: all the hardware is properly operating and your network is properly configured. Only that red-reset-button required me to run the stairs up and down between my desktop and the lab ;) and this is where i heard of enptool binary which can be used to reset the card from the PCI bus.

so get http://ixp2xxx.sourceforge.net/uengine/uengine-0.0.37.tar.gz (or a more recent one, if any), compile enptool.c (see command line history below) and run ./enptool reset as root to experience the magic ...

note: i enabled pretty much things about PCI access without exactly knowing what i did using the fconfig command in RedBoot, so if your ENP card does nothing when ./enptool reset is invoked, i suggest you read back chapter 6 of the programmer's guide, and fconfig your card -- which in turn will allow you to automate the linux image loading -- or at least won't require you to hit CTRL+C after 2 seconds everytime you reset :P

Now, your linux kernel was probably not that happy on the card: it doesn't have its root filesystem (with /dev, /etc, /bin and all the others). This is where the MontaVista distribution really gets useful.

Looking up port of RPC 100003/2 on 139.165.222.102                                 Root-NFS: Unable to get nfsd port number from server, using default Looking up port of RPC 100005/1 on 139.165.222.102                 Root-NFS: Unable to get mountd port number from server, using default mount: server 139.165.222.102 not responding, timed out              Root-NFS: Server returned error -5 while mounting /extra/target VFS: Unable to mount root fs via NFS, trying floppy.           VFS: Cannot open root device "nfs" or unknown-block(2,0) Please append a correct "root=" boot option             Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)

ooh, nasty me ... i disabled the NFS server, making the linux on ENP card rootless again.

If you have montavista installed somewhere, copy the .../arm/.../target directory somewhere more convenient (/extra/target in my case) and make sure you have it exported by NFS (that should be as easy as enabling NFS server from your favourite Linux management tool on the host and then edit /etc/exports to include /extra/target with radisys's favourites configuration)

If not, get at least busybox, the all-in-one shell, and make yourself ready to build a minimal filesystem step by step in /extra/target.

README: BusyBox has been written with size-optimization and limited resources in mind, both to produce small binaries and to reduce run-time memory usage. Busybox is also extremely modular so you can easily include or exclude commands (or features) at compile time. This makes it easy to customize embedded systems; to create a working system, just add /dev, /etc, and a Linux kernel.  Busybox (usually together with uClibc) has also been used as a component of "thin client" desktop systems, live-CD distributions, rescue disks, installers, and so on.

With just the appropriated commands in /extra/target/dev, you should be able to see the prompt on the minicom device. Once you have the console rerouted to the minicom interface, you've almost won the battle. Detailed commands are available in the faq-to-be.

The MontaVista linux also provides a web server with stats and telnet login (just in case the red bar from the minicom gets on your nerves :*) This one is a little more tricky to get working and requires that:

1. your kernel has support for Unix98 pseudo-terminals. If you get your kernel from ixp2xxx.sf.net, it's enabled.
2. you have /usr/sbin/in.telnetd properly set up in inetd configuration (if you're running a MontaVista filesystem, that should be the case, if not add telnet  stream  tcp     nowait  root    /usr/sbin/in.telnetd in.telnetd to /etc/inetd.conf
3. you have a place where /dev/pts/xx (slave pseudo-terminals) can be created and the /dev/ptmx (pseudo-terminal multiplexer) device for creating them.
4. you have mounted /dev/pts with devpts filesystem.
   

On a 2.4.xx kernel

The Radisys support package provide scripts (to be run on the embedded MontaVista linux) to get microengines, spi-3 (the FPGA bridge between IXP2400 and the ethernet controllers) and pm-3386 (the gigabit ethernet controllers proper) drivers loaded. The actual modules invoked are libossl.o, halMeDrv.o, halMev2_lib.o, pm338x.o and spi3br.o. While pre-compiled modules are provided (normally installed in the /opt/enp-2611 directory of your target system), they will be of no help on a home-compiled kernel (especially since ixp2xxx project provides 2.6.xx kernels and that Radisys's precompiled drivers are for 2.4.20).

The sources of these drivers have been installed by the java-installshield tool under
/opt/ixa_sdk_4.0/enp-2611/drivers/src

Note that the build script may be a bit annoying with your actual setup (where each package -- and especially the cross-compiled kernel sources -- are installed). Note, too, that the provided sources apparently contain 2.4.xx specific stuff such as MOD_INC_USE_COUNT.

On a 2.6.xx kernel

Instead, there's a patch ixp2000-netdev-0.2-2.6.15.diff that you can apply to a 2.6.15 kernel to add drivers support. You can retrieve the patch from ixp2xxx.sf.net and start your own compilation or wait for some pre-compiled ENP modules.

Disclaimer: these files are built from ixp2xxx.sf.net sources and kernel.org vanilia kernel. Refer to those sources for licensing issues. Moreover, by downloading any binaries from this site, you use them at your own risk.

In order to load succesfully the gigabit ports drivers, you need:

• a 2.6.xx kernel with module supports compiled in and which declares SRAM (and other registers) that the modules use. http://ixp2xxx.sourceforge.net/kernel/netdev/patch-2.6.15-rc1-ue1 do provide both the modules and SRAM, so you're suggested to patch your kernel with that rather than with ixp2000-netdev-xxx patch above. You can also grab home-compiled kernel here with support for modules and ENP SRAM compiled straight in by your humble servant.
• The enp2611_mod.ko itself, which you'll have to enable by hand when compiling the kernel. You can grab it here too.
• A busybox tool that has support for insmod, lsmod and other similar operations. Yeah, you can grab it too ;)
  

Install zImage-srammods in your tftp directory, enp2611_mod.ko into "/lib/modules" from your NFS-exported target filesystem and "my_busybox" wherever you want in your exported filesystem. You may not want to discover that "my_busybox" is not featuring as many applets as MontaVista linux do, so i suggest you only replace /sbin/insmod, /sbin/modprobe, /sbin/rmmod and /sbin/lsmod with symlinks to the my_busybox rather than replacing /bin/busybox itself.

• cd /sbin
• rm insmod modprobe rmmod lsmod
• ln -s /my_busybox insmod
• ln -s /my_busybox modprobe
• ln -s /my_busybox rmmod
• ln -s /my_busybox lsmod
• inmod /lib/modules/enp2611_mod.ko

If you properly followed the above instructions, you should now see messages indicating that the Media Switch Interface (MSF) ethernet driver is loaded, MAC addresses of eth1, eth2 and eth3 plus a warning saying that both three interfaces are currently down.

You can now activate the interfaces with e.g.

    ifconfig eth2 192.168.1.1 netmask 255.255.0.0
You should then see a message saying that your NIC Link is up. running ifconfig will confirm you your configuration.

>_<    Insmod: ELF file not for this architecture

Even with your own kernel and modules compiled, the "insmod" tool goes on with complains. That's because the busybox binary shipped by MontaVista does not know of 2.6.xx kernels and thus it doesn't want to load our 2.6.xx modules even on a running 2.6.xx kernel... This means we'll have to retrieve busybox package, configure it with (at least) support for module loading (including 2.6.xx), not forgetting that we want xscale_be- prefix in the "general build options > cross-compiler"

X_x    kernel panic at virtual address 0xfe900000

The enp2611 module expects that the kernel itself has declared the virtual memory mappings to access microengine registers aswell as SRAM channel. If this is not the case (e.g. wrong patch applied), the module still tries to initialize microengines to receive and transmit packets, but since the mapping is missing, you get a fault. If you compiled the enp2611 driver straight into the kernel, this can even prevent your kernel from booting.

0_o    I can ping GigE address even when no cable is plugged !?

This may happen if you plugged both the "management" port and the gigabit ethernet ports on the same LAN. In that case, ARP requests for the Gige address (say 139.165.222.106) is received on the management port aswel. The policy of linux kernel is then to reply to an ARP request if any of its interfaces has the requested IP address, which leads to the problem known as ARP flux. Plus, the ARP cache of the requestor will keep the latest received reply (likely to be the MAC address of the management port, thus, since it is only connected at 100Mbps). Yet, using arping -b 139.165.222.106, you will be able to check that your GigE ports work properly (e.g. by comparing values in /proc/net/dev).

While there exist workarounds (arp_filter, for instance), the best one would still to plug your GigE interfaces to different networks. Host 192.168.1.2 connected to GigE interface 192.168.1.1 knows that the management address 139.165.222.104 isn't attached to the same LAN, so it will not even try to send an ARP packet for 139.165.222.104 on 192.168.1.x :P

The ENP driver gives us a good example of how the microengines can be programmed to work with the Linux kernel. We can identify SRAM state initialization and microcode loading in ixpdev.c:


    /** code snippet from Lennert Buytenhek's driver **/

    printk(KERN_INFO "* preparing ring registers\n");
    /* 256 entries, ring status set means 'empty', base address 0x0000.  */
    ixp2000_reg_write(RING_RX_PENDING_BASE, 0x44000000);
    ixp2000_reg_write(RING_RX_PENDING_HEAD, 0x00000000);
    ixp2000_reg_write(RING_RX_PENDING_TAIL, 0x00000000);

    /* 256 entries, ring status set means 'full', base address 0x0400.  */
    ixp2000_reg_write(RING_RX_DONE_BASE, 0x40000400);
    ixp2000_reg_write(RING_RX_DONE_HEAD, 0x00000000);
    ixp2000_reg_write(RING_RX_DONE_TAIL, 0x00000000);

    for (i = 0; i < RX_BUF_COUNT; i++) {
        ixp2000_reg_write(RING_RX_PENDING,
            RX_BUF_DESC_BASE + (i * sizeof(struct ixpdev_rx_desc)));
    }

    printk(KERN_INFO "* load microengine[0]");
    ixp2000_uengine_load(0, &ixp2400_rx);
    ixp2000_uengine_start_contexts(0, 0xff);

Only two of the 8 microengines are required for this very simple functionnality. Microengine 0 will be in charge of receiving packets from GigE chips through the Media Switch interface, and microengine 1 will be transmitting packet of the kernel to the GigE chips. For both, packets are stored in DRAM (that is, where Linux kernel stands too) and RING_?X_* symbols are memory-mapped "scratch" registers.
The helper functions "uengine_load" and "uengine_start_contexts" are machine-specific functions provided by the kernel in arch/arm/mach-ixp2000/uengine.c, while ixp2000_reg_write() is simply a macro for writing to memory-mapped items of the IXP such as scratch pad, SRAM, etc.

The microcode to be loaded in the microengines has been "embedded" into the C module by the mean of list_to_header.pl which can be found in the uengine tarball on ixp2xxx.sf.net. The script operates on the .list file generated by intel's microengine assembler uca. E.g.

ixp2400_tx.uc source code; (C) 2004, 2005 Lennert Buytenhek	.reg @channel     .reg @packet_offset     .reg zero     immed[zero, 0]      /* initialize only in context 0 */     .begin         br!=ctx[0, mpacket_tx_loop#]     .end
ixp2400_tx.list file, corresponding bits. Note the microwords encoding in bold.	.0 F000000700 common_code         .import_var __chip_id i$__chip_id __chip_revision i$__chip_revision // ...                .reg @l0000!channel                 .reg @l0000!packet_offset                 .reg l0000!zero                 immed[l0000!zero, 0] .1 D803000011 common_code                 .begin                         br!=ctx[0, mpacket_tx_loop#] .2 3C400004E0 common_code                 .end
ixp2400_tx.ucode file, ready to be integrated in the kernel ...	static struct ixp2000_uengine_code ixp2400_tx = {     .cpu_model_bitmask    = 0x000003fe, // ...     .initial_reg_values    = (struct ixp2000_reg_value []) {         { -1, -1 }     },     .num_insns        = 77,     .insns            = (u8 []) {         0xf0, 0x00, 0x00, 0x07, 0x00,         0xd8, 0x03, 0x00, 0x00, 0x11,         0x3c, 0x40, 0x00, 0x04, 0xe0,         0x81, 0xf2, 0x02, 0x01, 0x00,         0xd8, 0x00, 0x80, 0x01, 0x00,         0xb0, 0x08, 0x06, 0x00, 0x00,

Loading and Runninng UOF files

The driver offered by Lennert is just that: a driver. While it has microcode loading facilities for its own purposes, it will not help you to load your microcode into the microengines -- except if you're happy with directly patching the _tx.uc and _rx.uc files. If you want to use the full power of intel's IXA SDK, you should instead use the device drivers provided by intel in /opt/ixa_sdk_.../me_tools/*. These are made of two parts: a user-side library and a kernel-side driver, glued together by the /dev/medrv0 device.

In addition to medrv0 device, application written for the IXA SDK typically use /dev/spi3br and /dev/pm338xx devices to program the gigabit ethernet hardware of the ENP card.

/dev/MeDrv0

This device is normally operated by the IXA SDK user-side libraries to communicate with the kernel-side code. It mainly provides mmap support (which will map the microengine portion of DRAM in your user process) and ioctl (for e.g. reporting memory sizes to user-level, poll/disable interrupts).
Sources offered by intel are of course aimed at MontaVista Linux 2.4.xx kernel. ENP-2611 linux support package also assume MontaVista 2.4.xx pro package and thus doesn't really help in building them.

You can find a patch against IXA-SDK 4.2 that worked pretty fine for me or even the precompiled halme+ossl+uclo driver for linux 2.6. Note that:

• the IXA come with three module, here you have a single-module driver;
• I shamelessly screwed up the build process to like and understand it better;
• There's a redundant memory map setting between the enp-enabled kernel proper and the IXA driver;
• I still have tests to drive before i can claim this patched driver to work properly.
  

And, of course, you'll need mknod MeDrv0 c 10 244 before you can work with the driver :P

User-side UOF Loader

This is the simplest part. using properly compiled libraries (that is, uclo.a, halMev2.a, libossl.a, rs_cntl.a, utils.a and dbgMe.a), you can load the UOF file and start the threads on microengines with the following code provided by tomas on the ixp2xxx mailing list:

#include <uclo.h>
#include <hal_mev2.h>
#include <halMev2Api.h>
#include <stdio.h>
int main() {
  unsigned int meMask=0xffffffff;

  UcLo_InitLib();
  UcLo_InitLibUeng(meMask);

  if ((ret = UcLo_LoadObjFile(&objHandle, ".uof file")) != UCLO_SUCCESS) {
    printf ("*** load failed *******\n");
    exit (1);
  }

  if (UcLo_WriteUimageAll(objHandle) != UCLO_SUCCESS) {
    printf ("*** write image failed\n");
    exit (1);
  }

  if (UcLo_VerifyUengine(objHandle,0) != UCLO_SUCCESS) {
        printf ("ME 0 code not matched\n");
  }


  halMe_Init(meMask);

  halMe_Start(0, 0xff); // starting ME0, all threads
  exit(0);
}
Note that if your code spans on multiple microengines, multiple call to halMe_Start() will be required with appropriated microengines numbers. The microcode-loader library will report the IDs of the microengines that have been loaded on console, from there, it's up to you to know which one should be started and which shouldn't.

# /opt/myloader /opt/enp2611_esp.uof Taking/read MSF out of reset... halMe_SetKernelBarrierVAddr halMe_SetKernelBarrierVAddr Checking UOF compatibility - ProdType=2, ProdRev=16, cpuType=0x2, minCpuVer=1, maxCpuVer=255 @@ loading Me=0x0, page=0, old_page=-1 @@ loading Me=0x1, page=0, old_page=-1 @@ loading Me=0x2, page=0, old_page=-1 @@ loading Me=0x3, page=0, old_page=-1 @@ loading Me=0x10, page=0, old_page=-1 @@ loading Me=0x11, page=0, old_page=-1 @@ loading Me=0x12, page=0, old_page=-1 @@ loading Me=0x13, page=0, old_page=-1 halMe_SetKernelBarrierVAddr #

ENP-Specific devices: /dev/spi3br and /dev/pm338xx

Examples provided by the IXA SDK aswell as demonstration provided with the ENP SDK are oriented around the idea of a system application that takes the control of all your board's resources (microengine, GiGE controllers, SPI3 bridge) for a given purpose. The code for this application will e.g. issue proper IOCTL calls to enable reception/transmission by the gigabit ethernet chips, or prepare SRAM/SDRAM values according to the expectations of the code running on microengines.
Most of the time, the UOF file to be loaded by a system application is simply hardcoded and the system application can be used to load/unload it, aswel as it can report/configure various parameters of the running application.

Like in the case of HalMev2 driver, ENP drivers (located under $IXA/enp-2611/drivers/) are specific to linux 2.4, but hopefully, they're way smaller than the microengines driver and easier to port to 2.6.xx . It should be noted, however, that these devices expect their major numbers to be given as parameters. Radisys SDK comes with helper scripts spi3br_load and pm338x_load that will take care of giving the proper major/minor numbers to /dev/spi3br, /dev/pm338x0 and /dev/338x1 and load the modules with the appropriate <driver>_major values.

In case of trouble with a system application (obscure error messages, etc), it's still possible to debug from the host it as it runs on the target. It mainly requires you to recompile the application with -g flag, to invoke it with gdbserver host:12345 <app> <params> and then to run xscale_be-gdb <app> on the host. Both gdbserver and xscale_be-gdb are provided with the montavista linux distribution, but they're just the "good old" tools compiled for ARM support.

You can connect to the waiting application by typing target remote <target>:12345 and then debug the application just like you would debug any application, with the exception that the program is already running (you should thus use continue rather run command in gdb). Another good news is that any graphical front end to gdb will happily work too if you take care to symlink "gdb" to xscale_be-gdb in your local directory ^_^

0_o UCLO error 820C (wrong UOF format)

Chances are that you're trying to load a .UOF file compiled for IXA SDK 4.0 with a loader/driver from IXA SDK 4.2. Radisys-provided "static forwarding application", for instance, will try to load enp2611_static_fwd.uof found in /opt/enp2611 (on the target), but the build utilities will ususally copy the pre-compiled one. Since you're running 4.2 tools (at least if you got them here) and that Radisys compiled the .uof with 4.0, it will fail.

Retrieve the sources of the .UOF (in $IXASDK/enp-2611/sample-application/Microcode in this case) and at least re-link them (recompiling from the .uc files couldn't be bad too).

X_x kernel panics after the drivers are removed

There's apparently a difference between the way register_chrdev() returns value between 2.4 and 2.6 kernels that makes ENP-specific drivers fail to unregister their name properly if the name has been given on the command line. That means trying to list all the character devices (e.g. when doing cat /proc/devices) will fail miserably.

0_o system application says "cannot open device"

Sounds like there's a missing module. Check what you've loaded, what devices are declared in /dev and whether their major/minor numbers match those in /proc/devices. If you used spi3br_load and pm338x_load tools, that shouldn't occur. If you see pm338x or spi3br bound to major number 253 or 254 in /proc/devices, that's another sign the modules haven't received their parameters as expected and are running with kernel-allocated which may be harder to guess.