Reference All In One

The number of different executable file types is as many and varied as the number of different image and sound file formats. Every Operating System seems to have several executable file types unique to itself. This part of the FAQ will give a brief rundown on the various types you will come across.

A quick intro to a few terms:

• TEXT is the actual exectuable code area,
• DATA is "initialised" data,
• BSS is "un-initialised" data.

The BSS (Below Stack Segment) needn't to be present in an executable file. At load-time, the loader will still allocate memory for it and wipes this memory with zeroes (this is assumed by C programs, for instance).

If you're looking for comprehensive informations, consider using the Programmer's File Format Collection and the Linkers and Loaders online book... You can also check Pierre's Library

EXE (DOS "MZ")

DOS-MZ was introduced with MS-DOS (not DOS v1 though) as a companion to the simplified DOS COM file format. DOS-MZ was designed to be run in real mode and reflects this, having a relocation table of SEGMENT:OFFSET pairings. A very simple format that can be run at any offset, it does not distinguish between TEXT, DATA and BSS. Since it was designed to run in real mode, its maximum filesize of code + data + bss is 1mb in size.

Operating Systems that use it: DOS, Win*, Linux DOS Emu, Amiga DOS Emu

EXE (Win 3.xx "NE")

The WIN-NE executable formated designed for Windows 3.x was the "NE" New-Executable. Again, a 16bit format, it alleviated the maximum size restrictions that the DOS-MZ had.

Operating Systems that use it: Windows 3.xx

EXE (OS/2 "LE")

The "LE" Linear Executable format was designed for IBM's OS/2 operating system by Microsoft. Supporting both 16 and 32bit segments.

Operating Systems that use it: OS/2, Watcom Compiler/Extender (DOS)

EXE (Win 9x/NT "PE")

ELF

A.OUT

A.OUT is the "original" binary format for Unix machines. It is considered obsolete today because of several shortcomings. However, as it is extremely simple and supported by many compilers/assemblers, it may be a good choice if you're willing to develop your own format or have more information than 'raw binary' for your bootloader.

Tell me about Filesystems

Filesystems are the machine's way of ordering your data on readable and/or writable media. They provide a logical way to access the stuff that you have down on disk so that you can read or modify extit. Which file system you use depends upon what you want to do with it. For example, Windows uses the Fat32 or NTFS filesystem. If your disk is really huge, then there's no point using Fat32 because the FAT system was designed in the days when nobody had disks as big as we do now. At the same time, there's no point using a NTFS filesystem on a tiny disk, because it was designed to work with large volumes of data - the overhead would be pointless for, say, reading a 1.44m floppy disk.

There are many different kinds of filesystems around, from the well-known to the more obscure ones. The most unfortunate thing about filesystems is that every hobbyist OS programmer thinks that the filesystem they design is the ultimate technology, when in reality it's usually just a bad copy of DOS FAT with a change here and there. The world doesn't need another crap filesystem. Investigate all the possibilites before you decide you roll your own.

FAT And its Variants

File Allocation Table (FAT) was introduced with DOS v1.0 (and possibly CP/M), supposedly written by Bill Gates. FAT is a very simple filesystem which is nothing more than a singular linked list of clusters. FAT filesystems use very little memory and is one of, if not the, most basic filesystem in existance today.

There are two versions of this simplified FAT, FAT12 and FAT16. FAT12 was designed for floppy disks and can manage a maximum size of 16mb using 12bit cluster numbers. FAT16 was designed for early hard disks and could handle a maximum size of 64kb * cluster_size. The larger the hard disk, the larger the cluster size would be, which lead to large amounts of "slack space" on the disk.

FAT12+FAT16 filesystems have fixed size for filenames of "8.3" and limited support for file attributes. You could also read the FAT12 document. You could also check out the FAT tutorial reported by Kemp.

VFAT

VFAT is an extension of FAT16 and FAT12 that has the ability to use long filenames (up to 255 characters i think). First introduced by Windows95, it uses a "cludge" whereby long filenames are marked with an "volume label"; attribute and filenames are subsequently stored in the 8.3 format in sequential directory entries. (This is a bit of an oversimplification, but close enough).

FAT32

FAT32 was introduced to us by Windows95-B and Windows98. FAT32 solved some of FAT's problems. No more 64kb max clusters! FAT32, as its name suggests, can handle a maximum of 4gig clusters per partition. This enables very large hard disks to still maintain very small cluster sizes and thus reduce slack space between files.

Is FAT32 really able to handle 4G clusters ? last time i looked to a FAT table, it looked to have last 4 bits cleared all the time, including the 'end of chain' tag, giving me the feeling that it was somehow more a "FAT28" than "FAT32" -- PypeClicker

Correct - FAT32 is actually only FAT28. Top 4 bits are currently "Undefined" (according to Microsoft's specification). Note that this does not mean 'top four bits should be "0"' - they should be honored. This means that if you need to alter a FAT entry, and the top 4 bits contain 1001, you should write it back to disk containing 1001 and not 0000. -- djhayman

Inode-based File Systems

Inodes (information nodes) are a crucial design element in most Unix filesystems: Each file is made of data blocks (the sectors that contains your raw data bits), index blocks (containing pointers to data blocks so that you know which sector is the nth in the sequence), and one inode block.

The inode is the root of the index blocks, and can also be the sole index block if the file is small enough. Moreover, as unix filesystems support hard links (the same file may appear several times in the directory tree), inodes are a natural place to store metadata such as file size, owner, creation/access/modification times, locks, etc.

HPFS (High Performace Filesystem)

The HPFS was designed by IBM/Microsoft for IBMs new windowing system, OS/2. It was designed to be fast, remove all the shortcomings of FAT, support long filenames, small cluster sizes, remove degfragmentation as much as possible and support more attributes.

HPFS is the precursor to NTFS and is, in a nutshell, NTFS minus the security features embeded into NTFS. Instead of storing cluster chains in a single linked list format, HPFS stores its information in sorted B-Tree's. This makes searching for files very fast.

Instead of keeping the directory tables and other descriptors at the start of the disk, HPFS bands them at regular intervals throughout the disk and in the middle of the disk, with the theory being that the heads only have to move half as much in any direction.

More information about HPFS can be found at: http://www.wotsit.org/download.asp?f=hpfs

NTFS (New Technology Filesystem)

NTFS is the native filesystem of WindowsNT. It is much like HPFS, but supports security features in the filesystem such as access control. Since WindowsNT is entirly unicode, NTFS is a unicode filesystem, each "character" being 16bits wide. NTFS adds quite a bit more to HPFS than just security features, though. First, it adds quite a bit of builtin redundancy -- with HPFS, wiping out one sector in the wrong place can render an entire volume inaccessible. Second, it adds support for multiple hard-links to a file (up 'til now, the only easy access has been via the POSIX subsystem, but NT 5/Win2K adds this to Win32 as well). Third, it supports an arbitrary number of file forks a la MacOS (except MacOS always has exactly 2 forks per file). Fourth, HPFS decrees that a cluster is always 512 bytes, and a cluster is always one sector. For the sake of performance and compatibility with some (especially Japanese) machines, NTFS allows sectors of other sizes. It also supports clusters of more than one sector, which tends to help performance a little.

NTFS is probably one of the most difficult file system to deal with, especially because of the lack of hacking experience and reliable documents about it. A read-only stable driver is in Linux source code base since kernel 2.4, while an experimental read-write driver is coming with linux 2.6.

The best information found about it so far is Andrew Tanenbaum's article. The Linux NTFS project also has some information about it, at http://linux-ntfs.sourceforge.net/ntfs/index.html. (Use the next / previous links at the top of the pages, or use the glossary.) You are welcome to add more.

ext2fs (Second Extended Filesystem)

The Second Extended Filesystem (ext2fs) was the default filesystem of Linux prior the advent of the journaling file systems ext3fs and ReiserFS. It has native support for UNIX ownership / access rights, symbolic and hard links and other Unix-native properties. Like HPFS, it tries to minimize head movement by distributing data across the disk. Also, by using "groups", it minimizes the impact of fragmentation. It is another "inode" based system. An ext2fs-partition is made up from blocks, which normally are 1K each. The first block (the bootblock) is zeroized, all the other blocks are divided into so-called block groups (normally, between 256 and 8192 blocks form a group). Each block group contains:

• a copy of the superblock (which is a mighty useful structure containing info about the filesystem);
• the filesystem descriptors (dunno what that is exactly)
• the block bitmap, tells which blocks are used
• the inode bitmap, tells which inodes are used (difference?)
• the inode table, which contains the inodes themselves
• the data blocks referenced by the inodes

The first inode is a special one; it is the bad blocks inode, which references all the damaged sectors of the partition. The fifth inode contains the bootloader, whereas the 11th contains the root directory.

Windows users can access ext2fs partitions with explore2fs.

Additional information about ext2fs:

• ext2-doc project: Second Extended File System - implementation-oriented documentation, describes internal structure in human language.
• Design and Implementation of the Second Extended Filesystem (overview)

ext3fs (Third Extended File System)

ext3fs is basically ext2fs with journaling added. If your ext3fs partition does not need journal replay, it can even be accessed with a 'simple' ext2fs driver.

ReiserFS

ReiserFS - homepage at http://www.namesys.com - is a file system that is free if your OS is free, and comes at a licensing cost if your OS is not free. It has excellent performance on large directories and small files, using "dancing trees" instead of B-trees, and does meta-data journaling to improve file-system stability across system crashes.

Unlike "classic" filesystems, Reiser allows you to have files that occupy less than one sector on the disk (i.e. it can store several tiny files or tails of files on the same sector) through its tree organization.

As of version 3.6, ReiserFS supports:

• max number of files - 232 - 3 => 4 G - 3
• max number files a dir can have - 232 - 4 => 4 G - 4, but in practice this value is limited by hash function. r5 hash allows about 1 200 000 file names without collisions
• max file size - 260 - bytes => 1 E, but page cache limits this to 16 T on architectures with 32 bit int
• max number links to a file - 232 => 4 G
• max filesystem size - 232 (4K) blocks => 16 T

Network-based File Systems

All these file systems are a way to create a large, distributed storage system from a collection of "back end" systems. That means you cannot (for instance) format a disk in 'NFS' but you instead mount a 'virtual' NFS partition that will reflect what's on another machine. Note that a new generation of File Systems is under heavy research, basing on latest P2P, cryptography and error correction techniques (such as the Ocean Store Project or Archival Intermemory.

NFS

NFS was invented by Sun Microsystems. It became widespread largely because it'a quite easy to implement. In return for its simplicity, it tends to give relatively poor performance and a nearly complete lack of safety. These are both largely due to its connectionless nature. When you request data from a file, the server sends you the requested information, but does NOT keep track of which clients have which files open. To keep you from seeing (terribly) out-of-date information from a file, the data you read has an "expiration date". If you refer to the data from more than, say, a minute, it will expire and your client will request the data from the server again, whether it's changed or not. If you write data to the file, you have no way of knowing whether somebody else has updated the information between your reading and writing your data, so you may overwrite things they've written with older data. To ensure at least a little bit of safety, the server is supposed to actually commit data you write to disk before it returns to you.

In other words, NFS works pretty well for read-only access to things like executables on a server. For things like on-line databases, it's essentially a disaster waiting to happen (and it usually doesn't wait very long).

More recent versions of the NFS spec have cured most of these problems, but support for these updates is still (years later) somewhat uneven.

AFS

AFS is the Andrew File System aka Advanced File System, similar to NFS to about the same degree that a tricycle is similar to a fighter jet -- they're both typically one-person vehicles. AFS is a drastically more robust design than NFS, and is intended for MUCH larger networks. OTOH, it's also much more difficult to implement completely -- to the point that it's not likely to be of much interest to most hobbyists and such writing a new OS.

RFS

RFS (Remote File System) was introduced in UNIX System V to compete with NFS and such. Unlike NFS, RFS is a connection-oriented system, so if, for example, two different machines access a file on a server, they get about the same semantics as if two processes on a single machine accessed the file. Note that NFS and RFS are both built on top of some sort of local file system, which determines things like inodes and such.

Unclassifiable and Other Stuff

BeFS

BeFS is the new filesystem for the Be Operating system. It is very much like the MacOS Filesystem, supporting multiple forks in a 64bit filesystem. One very useful feature it shares with the AmigaOS FFS is the ability for an application to set a "notify callback", i.e. being notified when a file or directory changes.

Get way more information about BeFS in Practical File System Design

FFS (Amiga)

The Amiga Fast File System, to put it bluntly, is not - or rather, it's fast only when compared to the OFS, the Original File System of AmigaOS 1.x.

There are many bright design ideas making the AmigaOS a very special thing, but the file system was not exactly part of it. It is prone to invalidation, holds redundant data, and its directory structure is comparatively slow to traverse. It also lacks any concept of multi-user environments.

Perhaps the only good thing with the Amiga FFS was the concept of the Rigid Disk Block (RDB) - a special area at the beginning of a disk, holding not only the partitioning information. It was also possible to store a file system there - a module that would tell a different AmigaOS machine how to read a partition if it was not formatted in FFS format but something else.

For those interested in its internals should try to find a copy of "The Amiga Guru Book" by Ralph Babel, which holds a complete reference of its rather complex block structure. (It also has a complete reference of the DOS library, as well as interesting information on various internals of the Amiga architecture. It is long out of print, but perhaps you can still find copies on eBay.) The old FAQ also held some info in the internals, which are preserved in the AmigaFFS Document.

FFS / UFS (BSD)

Not to be confused with the Amiga FFS, the BSD FFS / UFS is commonly used on hard disks for the *BSD and derivatives. What is usually called a "partition" is called a "slice" in *BSD, which is in turn subdivided into "partitions" - a naming pattern that leads to some confusion, and to rather cryptic device names (ad0s1c for the third partition on the second slice on the primary master ATAPI hard drive...).

XFS

XFS is Silicon Graphics "Next Generation Journalled 64-Bit Filesystem With Guaranteed Rate I/O" designed for IRIX based systems. XFS uses the standard inodes, bitmaps and blocks, and is compatable with EFS and NFS filesystems.

According to the XFS white paper it has;

• Scalable features and performance from small to truly huge data (petabytes)
• Huge numbers of files (millions)
• Exceptional performance: 500+ MBytes/second
• Designed with log/database (journal) technology as a fundamental part not just an extension to an existing filesystem
• Mission-critical reliability

BFS

BFS (UnixWare Boot File System) is a SCO specification for a KISS filesystem used at bootstrap. It only offers one directory and, due to the way information about blocks are stored, only one file opened for writing at a time.

http://www.penguin.cz/~mhi/fs/bfs/bfs-structure.html

From what i see, it also means BFS will have to do nasty things if a file must be extended after some other file has been created -- PypeClicker

Agreed, but it's not a general-purpose filesystem. One tends not to extend things like the kernel image or modules. -- Strib

The Microsoft version might have information left out in this document. Make sure you read Long Filenames Specification too.

Note that the FAT filesystem is covered by software patents.

File Allocation Table (FAT 12)

This paper concentrates on the FAT12 system only. It is broken down into several sections. Following a brief introduction on File Allocation Tables, the paper goes into a step by step instruction on how to read an MS-DOS File Allocation Table for a diskette (FAT12). The sections are in order:

• Introduction
• FAT12 (Diskette)
• Reading the Boot Sector
• Reading the Directory
• Finding the Beginning of the Boot, FAT, Directory, and Open Space
• File Allocation Table Entry Cluster Values
• Location of File in Open Space Area
• A printed copy of the file that is used to dissect the FAT12 table
• A printed copy of the Boot Sector and Directory
• A printed copy of the File Allocation Table
• A printed copy of the Beginning of the File in the Open Space Area
• A printed copy of the Ending of the File in the Open Space Area

In the Introduction, what is called the data area is called the Open Space Area later in the instructional part of the paper. And finally although, this paper does go into quite a bit of detail it is by no means complete.

Introduction

The File Allocation Table (FAT) is a table stored on every hard or floppy disk that indicates the status and location of all data clusters that are on the disk. The File Allocation Table can be considered to be the "table of contents" of a disk. If the file allocation table is damaged or lost, then a disk is unreadable. In a file server the FAT data is sometimes kept in the computer RAM for quick access and is easily lost if the system crashes as the result of a power failure.

The File Allocation Table is maintained by the operating system that provides a map of the clusters (the basic unit of logical storage on a disk) that a file has been stored in. When you write a new file, the file is stored in one or more clusters that are not necessarily next to each other; they may be rather widely scattered over the disk. A typical cluster size is 2,048 Bytes, 4,096 Bytes or 8,192 Bytes.* The operating system creates a FAT entry for the new file that records where each cluster is located and their sequential order. When you read a file, the operating system reassembles the file from clusters and places it as an entire file where you want to read it.

The hard disk is physically arranged by cylinders, heads, and sectors, that is how it is addressed by the hardware controller and the ROM BIOS, which addresses it at a physical level. For the operating system and other programs, however, this is cumbersome, since the physical number of cylinders, heads, and sectors varies from disk to disk. It would be convenient to view the disk as simply a large continuous block of sectors with simple sequential addresses.

MS-DOS does, in fact, view the sectors on a disk as a one-dimensional array of sectors numbered from 0 to n-1, where n is the total number of sectors on the disk. It therefore must translate from the logical sector numbers to physical to physical cylinder-head-sector, or CHS addresses. In doing so, MS-DOS sequentially numbers all the sectors of head 0, cylinder 0, then all the sectors of head 1, cylinder 0, and so on for each head, and then repeats this for each cylinder, to the end of the disk.

Furthermore, MS-DOS logically divides this array of sectors into five distinct areas, which are, in the order they appear on the disk,

• The partition table,
• The boot record,
• The File Allocation Table (FAT),
• The root directory, and

• The data area.

  The first four areas of the disk, collectively called the system area, are used by MS-DOS to keep track of the contents of the disk. The largest area of the disk, the data area, is where all user files and data reside. MS-DOS uses a special numbering scheme for the area called cluster numbering which is in addition to, but independent of logical sector numbers.

  The boot record occupies one sector, and is always placed in logical sector number (LSN) 0, which is physically cylinder 0, head 0, sector 1, the first sector of the first head of the first cylinder on the disk. This is the easiest sector on the disk for the computer to locate when it begins running.

  The File Allocation Table (FAT) is an array of integers in which each element represents one cluster in the data area. For each cluster in the data area the corresponding entry in the FAT contains a code which indicates the status of the cluster. The cluster may be available for use, it may be reserved by the operating system, it may be unavailable due to a bad sector on the disk, or it may be in use by a file.

  MS-DOS maintains a hierarchical directory structure in which there is one entry for every file on the disk.

  Data area is where all user files and data reside.

FAT 12 (Diskette)

Boot Sector

Reading the Boot Sector

Bytes (0-2)

The first three bytes 6B 3C and 90 disassemble to JMP 003C NOP. The reason for this is to jump over the disk format information. Since the first sector of the disk is loaded into ram at location 0x0000:0x7c00 and executed, without this jump, the processor would attempt to execute data that isn't code.

Bytes (3 - 10)

The first 8 Bytes (3 - 10) is the version of DOS being used. The next eight Bytes 29 3A 63 7E 2D 49 48 and 43 read out the name of the version. The official FAT Specification from Microsoft says that this field is really meaningless and is ignored by MS FAT Drivers, however it does recommend the value "MSWIN4.1" as some 3rd party drivers supposedly check it and expect it to have that value. Older versions of dos also report MSDOS5.1 and linux-formatted floppy will likely to carry "mkdosfs" here. If the string is less than 8 bytes, it is padded with zeroes.

Bytes (11 - 12)

The next two Bytes (11 - 12), 00 and 02, is the number of Bytes per sector. The first thing you do when reading a pair of Bytes is reverse them to read 02 00. 0200 is the number of Bytes per sector in hexadecimal or 512 Bytes per sector in decimal.

Byte 13

Byte 13 is the number of sectors per allocation (cluster). In this case it is one.

Bytes (14 - 15)

These two Bytes, 01 and 00, indicate the number of reserved sectors. Again you must reverse the Bytes to 00 01. There is one reserved sector.

Byte 16

This is the first Byte of the second row and it indicates the number of FAT's on the diskette. There are two.

Bytes (17 -18)

This indicates the number of directory entries. Reversing the Bytes E0 00 to 00 E0 and converting the number to decimal we have 224 directory entries.

Bytes (19 - 20)

The total sectors in the logical volume. Reversing 40 and 0B to 0B 40 and converting the number to decimal we have 2880 sectors in the logical volume. If that value is 0, it means there are more than 65535 sectors in the volume, and the actual count is stored in "Large Sectors (bytes 32-35).

Byte 21

This Byte (F0) indicates the media descriptor type, which is here a 1.44MB floppy.

Bytes (22 - 23)

These two Bytes, 09 and 00, indicate the number of sectors per FAT. Reversing 09 and 00 to 00 09, we see that we have nine sectors per FAT.

Bytes (24 - 25)

Number of sectors per track. Reversing the Bytes 12 and 00 to 00 12, there are eighteen sectors per track.

Bytes (26 - 27)

These two Bytes indicate the number of heads or sides on the diskette. Reversing the Bytes 02 and 00 to 00 02, we see that there are two sides to the diskette.

Bytes (28 - 29)

Number of hidden sectors. Both Bytes read zero, no hidden sectors.

Byte 30 Start of bootstrap routine is zero.

Directory

Reading the Directory

We had a thread where it shows that root directory might not be that simple. It seems like we should read all entries, skipping entries marked as 'volume label' if any. -- PypeClicker

Bytes (0 - 10)

Starting with the Byte 2620, the first 11 Bytes (0 - 10) is the name of the file with extension. If the 11 byte string is PROCESSATXT, then the 8.3 filename is PROCESSA.TXT since the first 8 bytes of the string comprise the filename and the last 3 are the extension. If the filename is less than 8 bytes or the extension is less than 3, padding spaces are added, e.g. a file name of LOADER.RC would be encoded simply as "LOADER RC " (that's two spaces after LOADER and one after RC).

Byte 11

This Byte lists the attributes of the file. To read this you must convert the hexadecimal Byte to binary. In this case 20 (hex) is converted to 0010 0000. Each of the eight bits represents an attribute of the file. When a bit is on, indicated by a one, the file has that attribute. Starting with the right most bit, which is the zero bit and working over to the left most bit the 7th bit the attributes are; read only, hidden, system file, volume label, sub-directory, archive, and the last two bits the 6th and 7th bits indicate resolved. In this particular file it is the 5th bit that is on meaning that it is an achieve file.

Bytes (12 - 21)

These are the reserved Bytes.

Bytes (22 - 23)

These two Bytes, 4E and 7B, indicate the time the file was made. To retrieve the time reverse the Bytes to 7B 4E and convert to binary 0111 1011 0100 1110. The hour is read from the first five bits, the minutes are read from the next six bits, and the seconds are read from the last five bits. So our time Bytes are read like this 01111 011010 01110. Reading the Bytes; the hour is 15, the minutes are 26, and the seconds are 14. Important, the seconds must be multiplied by 2 to get the true second reading. So the time that the file was created was 15:26:28 military time or 3:26:28 PM.

Bytes (24 - 25)

These two Bytes, 96 and 26, indicate the date the file was made. To retrieve the date reverse the Bytes to 26 96 and convert to binary 0010 0110 1001 0110. The year is read from the first seven bits, the month is read from the next four bits, and the day is read from the last five bits. So our date Bytes are read like this 0010011 0100 10110. Reading the Bytes; the year is 19, the month is 4, and the day is 22. The number for the year must be added with 1980 to get the correct year the file was made. So the date that the file was made was April 22, 1999.

Bytes (26 -27)

These two Bytes, 02 and 00, indicate the entry cluster value for both the FAT and the Open Space Area. More about this in the last two sections (File Allocation Table Entry Cluster Values and Location of File in the Open Space Area).

Bytes (28 - 31)

These four Bytes 1B, 0C, 00, 00 indicate the size of the file. Reversing the Bytes to 00 00 0C 1B and converting the number to decimal the size of the file is 3099 Bytes.

Finding the Beginning of the Boot, FAT, Directory, and Open Space

Boot Sector

as stated in the introduction the Boot Sector is always placed in logical sector number (LSN) 0, 0000.

File Allocation Table (FAT)

The File Allocation Table begins after the Boot Sector. To find the starting Byte, find the length of the Boot Sector which is one sector multiplied by the number of Bytes per sector (Bytes 11 and 12 of the Boot Sector). The File Allocation Table begins at 0200 (hex).

Directory

The Directory begins after both the Boot Sector and the File Allocation Tables. To find the starting Byte, find the number File Allocation Tables on the diskette (Byte 16 of the Boot Sector) and multiply this number with the number of sectors per FAT (Bytes 22 and 23 of the Boot Sector). Add this number with the number of Boot Sectors (which is one) to give you the total number of sectors of both the FAT and Boot sectors. Multiply total number of sectors by the number of Bytes per sector (Bytes 11 and 12 of the Boot Sector) giving you the starting Byte of the Directory.

(2 * 9) + 1 = 19 sectors; 19 sectors * 512 Bytes/ sector = 9728 Bytes (decimal) or 2600 Bytes (hex). The start of the Directory is 2600.

Open Space

The Open Space begins after the directory. To find the beginning of the Open Space you need to find the size of the directory in Bytes and add that to the beginning Byte of the Directory (2600). To find the size of the directory multiply the number of directory entries (Bytes 17 and 18 of the Boot Sector) by the Bytes per directory entries which in this case is given at 32 Bytes/directory entry of data (decimal).

224 directory entries * 32 Bytes/ directory entry = 7168 Bytes (decimal) or 1C00 (hex)

1C00 Bytes (hex) + 2600 Bytes (hex) = 4200 Bytes (hex). The start of the Open Space is 4200.

File Allocation Table Entry Cluster Values

Starting with the entry cluster value (Bytes 26 and 27 in the Directory), find the values (02 and 00) and reverse them to read 00 02 (hex). The result being **2 (hex or decimal).

Because, this result, the value of 2 is the same for both hexadecimal and decimal converting to decimal is not necessary, just remember that this next step is in decimal. Multiply this number 2 by 1.5 giving the number 3 (decimal) or 3 (hex). Now, go to the File Allocation table and retrieve the 3rd (0203) and 4th (0204) Bytes.¨ Remember to start your counting from zero. Take the two Bytes 03 and 40 and reverse them to 40 03. Because 3 is a whole integer, AND the binary value of the hexadecimal number of 4003 (0100 0000 0000 0011) to the binary value of the hexadecimal number 0FFF (0000 1111 1111 1111). The result is **3 (hex or decimal).

Convert the result above to decimal (if necessary) and multiply by 1.5 giving 4.5. So now we extract the 4th (0204) and 5th (0205) numbers from the File Allocation Table which are 40 and 00. Reverse the hexadecimal numbers to read 00 40. Because 4.5 is not a whole integer we right shift 0040 to read 0004. The result being **4 (hex or decimal).

Multiply 4 (decimal) by 1.5 giving 6. Now we go to the 6th (0206) and 7th (0207) number in the File Allocation Table which are 05 and 60. Reverse the numbers to read 60 05. Because 6 is a whole integer, AND the binary value of the hexadecimal number of 6005 (0110 0000 0000 0101) to the binary value of the hexadecimal number 0FFF (0000 1111 1111 1111). The result is **5 (hex or decimal).

Multiply 5 (decimal) by 1.5 giving 7.5. Now we read the 7th (0207) and 8th (0208) numbers in the File Allocation Table which are 60 and 00. Reversing the numbers we have 00 60. Because 7.5 is a fractional number we right shift 0060 to read 0006. The result is **6 (hex or decimal).

Multiply 6 (decimal) by 1.5 giving 9. Reading the 9th (0209) and 10th (0210) numbers in the File Allocation Table which are 07 and 80. Reverse the numbers to read 80 07. Because 9 is a whole integer, AND the binary value of the hexadecimal number of 8007 (1000 0000 0000 0111) to the binary value of the hexadecimal number 0FFF (0000 1111 1111 1111). The result is **7 (hex or decimal).

Take the decimal result above and multiply by 1.5 giving 10.5. Read the 10th (0210) and 11th (0211) Bytes in the File Allocation Table, the numbers are 80 and 00. Reversing the numbers we have 00 80. Because 10.5 is not a whole integer we right shift 0080 to 0008. The result is **8 (hex or decimal).

• • This number in decimal form is used also to calculate the Location of File in Open Space Area (Next Section).

Multiply 8 (decimal) by 1.5 giving 12. Now extract the 12th (0212) and 13th (0213) Bytes in the File Allocation Table. These numbers are FF 0F. Reverse the numbers to read 0F FF. This value 0FFF (hex) indicates the end of this file.

Location of File in Open Space Area

To find the location of the file in the Open Space Area take the decimal results of the File Allocation Table entry cluster values, as denoted by the double asterisks, and subtract 2. Then multiply by the number of Bytes per sector, which is indicated in Bytes 11 and 12 in the Boot Sector. In this case the Bytes per sector value is 512 (decimal). Finally take this value in Bytes, convert it to hexadecimal, and add it onto the starting location of the Open Space Area, which in this case is 4200.

(2 - 2) sectors * 512 Bytes per sector = 0 Bytes (decimal) 0 Bytes (hex) + 4200 Bytes = 4200 Entry value of the first cluster is 4200.

(3 - 2) sectors * 512 Bytes per sector = 512 Bytes (decimal) 200 Bytes (hex) + 4200 Bytes = 4400 Entry value of the second cluster is 4400.

(4 - 2) sectors * 512 Bytes per sector = 1024 Bytes (decimal) 400 Bytes (hex) + 4200 Bytes = 4600 Entry value of the third cluster is 4600.

(5 - 2) sectors * 512 Bytes per sector = 1536 Bytes (decimal) 600 Bytes (hex) + 4200 Bytes = 4800 Entry value of the fourth cluster is 4800.

(6 - 2) sectors * 512 Bytes per sector = 2048 Bytes (decimal) 800 Bytes + 4200 Bytes = 4A00 Entry value of fifth cluster is 4A00.

(7 - 2) sectors * 512 Bytes per sector = 2560 Bytes (decimal) A00 Bytes + 4200 Bytes = 4C00 Entry value of sixth cluster is 4C00.

(8- 2) sectors * 512 Bytes per sector = 3072 Bytes (decimal) C00 Bytes + 4200 Bytes = 4E00 Entry value of seventh cluster is 4E00.

Links to more information about FAT

• http://scottie.20m.com/fat.htm
• from raw bits to directory listing (code posted) in the forum

1.1 Root Block

The root of the tree is the root block, which is at a fixed place on the disk. The root is like any other directory, except that it has no parent, and it's secondary type is different. AmigaDOS stores the name of the disk volume in the name field of the root block.

Each filing system blck contains a checksum, where the sum (ignoring overflow) of all the words in the block is zero.

          +---------------+
        0 |  T. SHORT     | Type
          |---------------|
        1 |       0       | header key (always 0)
          |---------------|
        2 |         0     | Highest seq number (always 0)
          |---------------|
        3 |   HT SIZE     | Hashtable size (=blocksize -56)
          |---------------|
        4 |       0       |
          |---------------|
        5 |   CHECKSUM    |
          |---------------|
        6 |     hash      |
          |     table     |
          /               /
          \               \
  SIZE-51 |               |
          |---------------|
  SIZE-50 |  BMFLAG       | TRUE if bitmap on disk is valid
          |---------------|
  SIZE-49 |   bitmap      | Used to indicate the blocks
  SIZE-24 |    pages      | containing the bitmap
          |---------------|
  SIZE-23 |    DAYS       | Volume last altered date and time
          |---------------|
  SIZE-22 |    MINS       |
          |---------------|
  SIZE-21 |    TICKS      |
          |---------------|
  SIZE-20 |     DISK      | Volume name as a BCPL string
          |     NAME      | of <= 30 characters
          |---------------|
  SIZE-7  |   CREATEDAYS  | Volume creation date and time
          |---------------|
  SIZE-6  |   CREATEMINS  |
          |---------------|
  SIZE-5  |  CREATETICKS  |
          |---------------|
  SIZE-4  |       0       | Next entry on this hash chain
          |---------------| (always 0)
  SIZE-3  |       0       | Parent directory (always 0)
          |---------------|
  SIZE-2  |       0       | Extension (always 0)
          |---------------|
  SIZE-1  |    ST.ROOT    | Secondary type indicates root block
          +---------------+

1.1.2 User Directory Blocks

          +---------------+
        0 |   T.SHORT     | Type
          |---------------|
        1 |   OWN KEY     | Header Key (pointer to self)
          |---------------|
        2 |       0       | Highest Seq Number (always 0)
          |---------------|
        3 |       0       |
          |---------------|
        4 |       0       |
          |---------------|
        5 |  CHECKSUM     |
          |---------------|
        6 |               |
          |    hash table |
          /               /
          \               \
  SIZE-51 |               |
          |---------------|
  SIZE-50 |    Spare      |
          |---------------|
  SIZE-48 |    PROTECT    |  Protection bits
          |---------------|
  SIZE-47 |       0       | Unused (always 0)
          |---------------|
  SIZE-46 |               |
          |   COMMENT     | Stored as  BCPL string
  SIZE-24 |               |
          |---------------|
  SIZE-23 |     DAYS      | Creation date and time
          |---------------|
  SIZE-22 |     MINS      |
          |---------------|
  SIZE-21 |    TICKS      |
          |---------------|
  SIZE-20 | DIRECTORY NAME| Stored as a BCPL string <=30 chars
          |---------------|
  SIZE-4  | HASHCHAIN     | Next entry with same hash value
          |---------------|
  SIZE-3  |    PARENT     | back pointer to parent directory
          |---------------|
  SIZE-2  |      0        | Extension (always 0)
          |---------------|
  SIZE-1  |  ST.USERDIR   | secondary type
          +---------------+

User directory blocks have type T.SHORT and secondary type ST.USERDIRECTORY. The six information words at the start of the block also indicate the block's own key (this is, the block number) as a consistency check and the size of the hash table. The 50 information words at the end of the block contain the date and time of creation, the name of the directory, a pointer to the next file or directory on the hash chain, and a pointer to the directory above.

To find a file or sub-directory, you must first apply a hash function to its name. This has function yields and offset in the hash table, which is the key of the first block on a chain linking those with the same hash value (or 0, if there are none). AmigaDOS reads teh block with this key and compares the name of the block with the required name. If the names do not match, it reads the next block on the chain, and so on.

1.1.3 File Header Block

           +------------+
        0  |   T.SHORT  | Type
           |------------|
        1  |   OWN KEY  | Header Key
           |------------|
        2  | HIGHEST SEQ| Total number of data blocks in file
           |------------|
        3  |  DATA SIZE | Number of data block slots used
           |------------|
        4  | FIRST DATA | First data block
           |------------|
        5  |  CHECKSUM  |
           |------------|
        6  |            |
           /            /
           \            \
           | DATA BLK 3 |
           | DATA BLK 2 | List of data block keys
  SIZE-51  | DATA BLK 1 |
           |------------|
  SIZE-50  |  Spare     |
           |------------|
  SIZE-49  |   PROTECT  | Protection bits
           |------------|
  SIZE-48  |  BYTESIZE  | Total size of file in bytes
           |------------|
  SIZE-46  |            |
           |  COMMENT   | Comment as a BCPL string
  SIZE-24  |            |
           |------------|
  SIZE-23  |    DAYS    | Creation date and time
           |------------|
  SIZE-22  |    MINS    |
           |------------|
  SIZE-21  |    TICKS   |
           |------------|
  SIZE-20  | FILE NAME  | Stored as BCPL string <= 30 chars
           |------------|
  SIZE-4   |  HASHCHAIN | Next entry with same hash value
           |------------|
  SIZE-3   |   PARENT   | Back pointer to the parent directory
           |------------|
  SIZE-2   |  EXTENSION | Zero pointer to the first extension
           |------------| block
  SIZE-1   |  ST. FILE  | Secondary type
           +------------+

Each terminal file starts with a file header block, which has type T.SHORT and secondary type ST.FILE. The start and end of the block contain name, time, and redundancy information similar to that in a directory block. The body of the file consists of Data blocks with sequence numbers from 1 upwards. AmigaDOS stores the addresses of these blocks in consecutive words downwards from offset size-51 in the block. In general, AmigaDOS does not use all the space for this list and the last data block is not full.

1.1.4 File List Block

If there are more blocks in the file than can be specified in the block list, then the EXTENSION field is non-zero and points to another disk block which contains a further data block list. The following figure explains the structure of the file list block.

           +-------------+
        0  |   T. LIST   | Type
           |-------------|
        1  |   OWN KEY   | Header Key
           |-------------|
        2  | BLOCK COUNT | =number of data blocks in block list
           |-------------|
        3  | DATA SIZE   | Same as above
           |-------------|
        4  | FIRST DATA  | First Data Block
           |-------------|
        5  |  CHECKSUM   |
           |-------------|
        6  |             |
           /             /
           \             \
           | BLOCK N+3   |
           | BLOCK N+2   | Extended list of data block keys
  SIZE-51  | BLOCK N+1   |
           |-------------|
  SIZE-50  |      info   | (unused)
           |-------------|
  SIZE-4   |     0       | Next in hash list (always 0)
           |-------------|
  SIZE-3   |   PARENT    | File header block of this file
           |-------------|
  SIZE-2   | EXTENTSION  | Next extension block
           |-------------|
  SIZE-1   |   ST. FILE  | Secondary type
           +-------------+

There are as many file extension blocks as required to list the data blocks that make up the file. The layout of the block is very similar to that of a file header block, except that the type is different and the date and filename fields are not used.

1.1.5 Data Block

           +-------------+
        0  |   T. DATA   | type
           |-------------|
        1  |   HEADER    | header key
           |-------------|
        2  |   SEQ NUM   | Sequence number
           |-------------|
        3  |  DATA SIZE  |
           |-------------|
        4  |  NEXT DATA  | next data block
           |-------------|
        5  |  CHECKSUM   |
           |-------------|
        6  |             |
           |             |
           |             |
           |             |
           |    DATA     |
           |             |
           |             |
           |             |
           +-------------+

Data blocks contain only six words of filing system information. These six words refer to the following:

• type (T.DATA)
• pointer to the file header block
• sequence number of the data block
• number of words of data
• pointer to the next data block
• checksum

Normally, all data blocks except the last are full (that is, they have a blocksize = blocksize-6). The last data block has a forward pointer of 0.

(Information taken from the Intel manuals to give an overview over the individual generation's capabilities.)

Intel Processors

These processors from Intel use the CPUID string "GenuineIntel"

Intel 386

Successor to the 80286, the Intel 386 is the first processor of the IA32 architecture. It has 32 bit wide registers, supports 4 kByte paging, and a flat memory model in addition to the segmented memory model of the 80286.

Intel 486

The 486 integrates a 80x87 FPU on-chip, and supports power saving functions (System Management Mode, StopClock, AutoHaltPowerdown).

Pentium

The Pentium supports 4 MByte paging in addition to the usual 4 kByte paging, integrates an APIC and (in later steppings) MMX SIMD registers (single instruction, multiple data). It also supports 2-way multiprocessing.

Pentium Pro

The Pentium Pro supports PAE (36 bit address space), but does not have the MMX registers of the Pentium.

Pentium II

The Pentium II again supports MMX (as well as PAE), as well as additional low-power states: AutoHALT, Stop-Grant, Sleep, and DeepSleep.

Pentium II Xeon

The Xeon supports 4/8/+ way multiprocessing.

Pentium III

The Pentium III supports SSE (128 bit packed single FP SIMD).

Pentium IV / Pentium M

The Pentium IV as well as the (mobile) Pentium M both support SSE2; the Pentium IV also supports Hyper-Threading (one-chip multiprocessing).

Advanced Micro Device Intel-compatible Processors

The biggest competitor to Intel at this time (2004 August). They came into being slightly after Cyrix with a 5k86 (being a 486 compatible similar to the 5x86, don't confuse them) and then followed it up by a K6 processor. This one was faster than the Pentiums, and more popular than the Cyrix ones because they both didn't rate it (afaik), and they didn't overheat (as was claimed, untrue, for the Cyrixes).

The CPUID identifier string is "AuthenticAMD"

K6

The K6 processor was a very nice processor, being Pentium compatible and doing anything the Pentium half could. It became impopular because it was too damn fast, it did a LOOPcc within 2 cycles, where the Pentiums took 18 cycles.

Because software couldn't handle the sheer speed, it made errors, and thus caused frequent "Blue Screen"s. Microsoft issued a K6 patch which was mandatory for all K6 users.

K6-2

AMD had a lesson learned there, don't make your processor too fast in some instructions. Not that they were put off by that, they just added 16 wait states to the execution of the LOOPcc and thus caused it to slow to the speed of a Pentium. AMD didn't just do this however. They added a special case (speculation, might be coincidence) for the DEC (E)CX; Jcc combination, which is semantically equivalent with the LOOPcc instruction, but this semantic equivalency and the loop being faster on Intels caused the loop instruction to always be used. Nobody used the DEC/Jcc combo. They kept the original speed for this combo and specified in their optimization manuals that this was the preferred method over the loopcc instruction.

It also featured a new technology, the 3DNOW! technology, which was MMX using floating point numbers, and multiplexed (again) on the floating point registers. The K6-2 was quite popular, and scaled higher than the P1 ever did. It was largely compatible with the P2, but (afaik) not completely.

K6-3

They started this design off with the concept of not making it underpowered in any place, and to make it at least P2 compatible. It was fully P2 compatible.

The K6-3 was not too popular, mainly because the K6-2 did very well and people didn't see why they should buy a more expensive K6-3 for the same amount of megahertz. This of course was a joke, same as it is to call a 2GHZ opteron slower than a 2.2GHZ celeron.

A little known fact about the K6-3 is that it is in fact an Athlon, minus a few instructions, and minus one very important piece. The K6-3 suffered from a bottleneck at the instruction decode unit (which converts the X86 instructions to native instructions). It could only handle 2 in a cycle, which it made during about 20-30% of the cycles for average software. For optimized software you could bring it to 100% easily, and still want another channel. This wasn't too weird, because it did have 3 execution units of each type (ALU / MMX / loadstore) which were not used much at all. Note that these units are units executing the native instructions, so making 3 of each is not a stupid idea. They needed a new front end, and of course a new copy of instructions from Intel.

Athlon (first try)

The first models of the Athlon were distinct, they were the first time that a competitor to Intel actually had a faster processor, without Intel having a backup plan. It was poised against the PIII, which at that time was their top model and best-running one too. The athlon beat them to the 1GHZ mark, and at that time the 1GHZ had become completely irrelevant. It just meant that they had a new size to mark their processors with. Intel missed the point here, and they did until very shortly ago. The GHZ myth had been broken, the Athlon at 1.1GHZ was still faster than the PIII at 1.3GHZ, and people knew. They didn't go for a P3 if a faster athlon was available at a lower clock speed, and at a lower price.

Athlon XP / MP / Duron (new style)

AMD switched to a big offensive, trying to persuade the buyers to demand AMD CPU's instead of being OK with Intels. The new versions of these processors were all just a tad better than the previous one, could do a slight number of instructions more (the Athlons started with not even SSE1, and from model 6 (both Athlon and Duron) they supported it). The processors also advanced very slightly in each other direction, making each new type just a tad faster than the previous one. In the end of the GHZ wars (past year, about) the fastest Athlon was running at 2.2GHZ, but outperformed the better half of the 3GHZ P4's.

AMD64 based CPU's

This is slightly offtopic here, but still quite relevant, since these processors all support the entire IA32 family natively. AMD created a new processor, with 64-bit (actually 48-bit, but who notices those 16 bits?) memory addressing and 64-bit calculations, being very compatible with the old style CPU's. So compatible, that the core for 32-bit and 64-bit is essentially equal, aside from the size of calculations and the support of a few encodings that were in effect redundant. They removed a few 1-byte opcodes (about 20 in total, including all 1-byte INC and 1-byte DEC instructions) to make place for a new REX prefix. They modified it to use 16 registers instead of 8, added a load of new names, got the old software working, and optimized the 32-bit performance to unprecedented levels. These CPU's outperform the P4 at any clock speed, in almost (1/20 programs not) any calculation-intensive program. This made them very popular, but also very expensive, The cheapest nowadays is around 180 dollars, or euro's.

Other CPU vendors making similar chips

Cyrix

Cyrix was a well-known CPU vendor from the 386 years (and slightly before) up to the Pentium II times, when it more or less vanished inside Via. Via now uses the name as a CPU name (not making it clearer), but this section is about the Cyrix CPU's. The processors supporting CPUID call it a "CyrixInstead"

Cyrix 387

This isn't actually a processor, but is the most famous Cyrix processor. It was the fastest coprocessor to the 386 to be found, and was even very usable aside a 486-SX. These were the main line of money for Cyrix.

Cyrix 5x86

A processor that performed as a 486 and was socket-compatible. Is not a pentium compatible, misses required instructions (such as cmpxchg8b).

Cyrix 6x86MX / M1

This processor is, even though the name suggests otherwise, compatible with the 586 (Pentium). It didn't contain any of the MMX or PPro features but is nevertheless very nice. It performed slightly better per cycle, and was thus given ratings. This was the time they were loathed for rating their processors.

Cyrix M2

Was a Pentium MMX compatible processor, also using ratings which gave it a bad name to start with. It was again socket-compatible to the Pentium MMX and the older Pentiums (without MMX). It supported a few features from the Pentium Pro, among which the very usable CMOVcc set. This however wasn't well known at the time, and nobody seemed to care.

There were possibly more but the current author can't recall which ones. Suffice to say, if there were any they were impopular and they were soon gone. The company was bought by Via.

Rise Technologies

I've only heard about this company making Pentium-compatible chips, without MMX, but I don't know any detail but the CPUID identifier string. It just stuck. The string was "RiseRiseRise", or the same in all 3 dwords (making a search for it very easy).

partially related thread: AT,XT and PC

Category: CollectedKnowledge, HardWareCpu

Virtual 8086 mode is a sub-mode of ProtectedMode. In short, virtual 8086 mode is whereby the cpu (in protected mode) is running a "Emulated" 16bit 'segmented' model (real mode) machine.

I don't enable V86 myself. Why should i care ?

The most common problem with v86 mode is that you can't enter ProtectedMode inside a v86 task. In other words, if you are running Windows or have emm386 in memory, you can't do a "raw" switch into protected mode (it causes an exception, iirc). DOS extenders worked around that problem using either VCPI or DPMI interfaces to switch into pmode (actually, promoting their V86 task as a 'regular' user task). For an OS programmer such interfaces are simply useless as they're part of another OS.

There are a few other more "technical" problems people have when using v86 mode, mostly because v86 has some instructions "emulated" by what's known as a v86-monitor program, as the cpu is in protected mode, some instructions are high up on the security/protection level and running those directly would cause no-end of trouble for the OS.

Such technicalities are beyond the scope of a simple FAQ. If you wish to learn more about virtual mode, i suggest you read the corresponding chapter of the HollyIntelManual.

How do i detect v8086 ?

EFLAGS.VM is NEVER pushed onto the stack if the V86 task uses PUSHFD. You should check if CR0.PE=1 and then assume it's V86 if that bit is set.

detect_v86:
        smsw    ax
        and     eax,1           ;CR0.PE bit
        ret

VM mode detection is mainly useful when writing DOS extenders or other programs that could be started either in plain real mode or in virtual mode from a protected mode system. An 'ordinary' bootloader shouldn't worry about this since the BIOS will not set up VM86 to read the bootsector ;)

I heard it could help me. How can i support it ?

Indeed, VM86 can be of high interrest if you need to access BIOS functions while you're in ProtectedMode. This is essentially useful in order to set up video mode. As many modern card/chipsets lack support for VBE3 protected mode interface, setting up a VM86 task that will perform the proper 'set video mode' call remains the method.

TimRobinson has provided a very nice tutorial about VM86 mode. BeyondInfinity also has a working implementation (combined VM86+VBE task). See VirtualMonitor page for more implementation considerations.

Argh! My kernel is below 1MB! what can i do ?

TimRobinson and many others suggests that you put your kernel at a 'high' logical address (e.g. 0xC0000000) to avoid VM86 tasks to interfere with it. This is especially important when your kernel is large and leaves no room for VM86 code below 1MB, or when you plan to run 'full programs' within your VM86 box.

If all you need is a BIOS interrupt wrapper, then you can easily do the following:

1. ensure that your 16bits code is on a separate page from any 32 bits code
2. enable paging
3. make kernel pages unwritable (and unreadable ?) for DPL3 and allow user-access only to those pages that contains your 16 bits code and pages that contains BIOS code or data.

Can i use VM86 for disk access ?

Theorically yes, though it is probably not a GoodIdea(tm), as most BIOS disk access will include IRQ handlers, DMA transfers which you can hardly control from your VM monitor, and may stick in VM86 task while the BIOS waits for an interrupt response while a 'good' driver would have let the CPU free for other processes.

Remember of your old MS9x system freezing when doing a disk access ? that was most of the time due to an INT13-through-VM86 problem.

Categories: FAQ, HardWareCpu

Related forum threads

Creating vm86 task VM86 and INT10h kernel location & VM86

add yours here

Additionnal links

add yours here

I wrote and tested this on my own K6 (k6-200) and it works ok, but I was unable to find anyone with a K6-2 (CXT core) or K6-3 since there is two different methods for enabling writeback mode. It should work fine on k6-2 CXT and K6-3 processors.

With some tweaking, can be put into anyone's OS.

You call AMD_K6_writeback with the CPUID results family, model and stepping, only when you are sure you have an AMD cpu.

Here's the code, using InlineAssembly

void AMD_K6_writeback(int family, int model, int stepping)
{
    /* mem_end == top of memory in bytes */
    int mem=(mem_end>>20)/4; /* turn into 4mb aligned pages */
    int c;
    union REGS regs;

    if(family==5)
    {
        c=model;

        /* model 8 stepping 0-7 use old style, 8-F use new style */
        if(model==8)
        {
            if(stepping<8)
                c=7;
            else
                c=9;
        }

        switch(c)
        {
        /* old style write back */
        case 6:
        case 7:
            AMD_K6_read_msr(0xC0000082, &regs);
            if(((regs.x.eax>>1)&0x7F)==0)
                kprintf("AMD K6 : WriteBack currently disabled\n");
            else
                kprintf("AMD K6 : WriteBack currently enabled (%luMB)\n",
                    ((regs.x.eax>>1)&0x7F)*4);

            kprintf("AMD K6 : Enabling WriteBack to %luMB\n", mem*4);
            AMD_K6_write_msr(0xC0000082, ((mem<<1)&0x7F), 0, &regs);
            break;

        /* new style write back */
        case 9:
            AMD_K6_read_msr(0xC0000082, &regs);
            if(((regs.x.eax>>22)&0x3FF)==0)
                kprintf("AMD K6 : WriteBack Disabled\n");
            else
                kprintf("AMD K6 : WriteBack Enabled (%luMB)\n",
                    ((regs.x.eax>>22)&0x3FF)*4);

            kprintf("AMD K6 : Enabled WriteBack (%luMB)\n", mem*4);
            AMD_K6_write_msr(0xC0000082, ((mem<<22)&0x3FF), 0, &regs);
            break;
        default:    /* dont set it on Unknowns + k5's */
            break;
        }
    }
}

void AMD_K6_write_msr(ULONG msr, ULONG v1, ULONG v2, union REGS *regs)
{
    asm __volatile__ (
        "pushfl\n"
        "cli\n"
        "wbinvd\n"
        "wrmsr\n"
        "popfl\n"
        : "=a" (regs->x.eax),
          "=b" (regs->x.ebx),
          "=c" (regs->x.ecx),
          "=d" (regs->x.edx)
        : "a" (v1),
          "d" (v2),
          "c" (msr)
        : "eax",
          "ecx",
          "edx",
          "ebx",
          "memory");
}

void AMD_K6_read_msr(ULONG msr, union REGS *regs)
{
    asm __volatile__ (
        "pushfl\n"
        "cli\n"
        "wbinvd\n"
        "xorl %%eax, %%eax\n"
        "xorl %%edx, %%edx\n"
        "rdmsr\n"
        "popfl\n"
        : "=a" (regs->x.eax),
          "=b" (regs->x.ebx),
          "=c" (regs->x.ecx),
          "=d" (regs->x.edx)
        : "c" (msr)
        : "eax",
          "ecx",
          "edx",
          "ebx",
          "memory");
}

Categories: CollectedKnowledge, HardWareCpu

This page tries to clear ideas about x86-64 CPUs (AMD64 and Intel's equivalent EM64T implementation). IA-64 (Itanium) are really a different beast and not addressed here.

Features

What does Long Mode offer ?

Long mode extends general registers to 64 bits (RAX, RBX, RIP, RSP, RFLAGS, etc), and adds an additional 8 integer registers (R8, R9, ..., R15) plus 8 more SSE registers (XMM8 to XMM15) to the CPU. Linear addresses are extended to 64 bit (however, a given CPU may implement less than this) and the physical address space is extended to 52 bits (a given CPU may implement less than this). In essence long mode adds another mode to the CPU

• Real mode
• Legacy mode (32 bit protected mode)
• Long mode (64 bit protected mode)
• System Management mode

Long mode does not support hardware task switching or virtual 8086 tasks, and most of the segment register details are ignored (a flat memory model is required). In long mode the current CS determines if the code currently running is 64 bit code (true long mode) or 32 bit code (compatability mode), or even 16-bit protected mode code (still in compatability mode).

The first 64 bit CPUs from both Intel and AMD will support 40 bit physical addresses and 48 bit linear addresses.

Setting up ...

How do I detect if the CPU is 64 bits ?

You can find that out by checking CPUID. All AMD64 compliant processors have the longmode-capable-bit turned on in the extended feature flags (bit 29) in EDX, after calling CPUID with EAX=0x80000001. There are also other bits required by long mode, but you can see those yourself in CPUID at AMD general purpose instruction reference

How do i enable Long Mode ?

The steps for enabling long mode are

• Disable paging
• Set the PAE enable bit in CR4
• Load CR3 with the physical address of the PML4
• Enable long mode by setting the EFER.LME flag in MSR 0xC00000080
• Enable paging

Now the CPU will be in compatability mode, and instructions are still 32-bit. To enter long mode, the D/B bit (bit 22, 2nd dword) of the GDT code segment must be clear (as it would be for a 16-bit code segment), and the L bit (bit 21, 2nd dword) of the GDT code segment must be set. Once that is done, the CPU is in 64-bit long mode.

Are there restrictions on 32 code running in Legacy Mode ?

x86-64 processors can operate in a legacy mode, they still start in real mode and protected mode is still available (along with the associated v8086 mode). This means an x86 operating system, even DOS, will still run just fine. The only difference is that physical addresses can be up to 52 bits (or as many bits as implemented by the CPU) when PAE is used.

However, there is nothing like Virtual8086 Mode (16 bits support) once in long/compatibility mode.

Can i enable Long Mode directly ?

Protected mode must be entered before activating long mode. A minimal protected-mode environment must be established to allow long-mode initialization to take place. This environment must include the following:

• A protected-mode IDT for vectoring interrupts and exceptions to the appropriate handlers while in protected mode.
• The protected-mode interrupt and exception handlers referenced by the IDT.

• Gate descriptors for each handler must be loaded in the IDT.

  --AMD64 docs, volume 2, section 14.4 (Enabling Protected Mode), 24593 Rev. 3.10 February 2005

That being said, we have a thread where Brendan shows how you can enable 64-bit long mode with no 32-bit IDT and no 32-bit segments ... Be assured, however, that any paging-related exception that occurs in long mode before you enable 64-bit IDT will cause the processor to reset due to a triple fault ...

64bit Environment Models

There are three 64bit programming models you need to consider; LP64, ILP64, LLP64, each mode has its own pitfalls. The I/L/P stand for Int, Long, Pointer, and the 64 means thats how many bits in each.

This LP64 means Longs and Pointers are 64bits wide. LL is a special case and means long-long...

DataTypes

This table lists the breakdown of sizes in the various programming models.

64bit OS Modes

The following table lists what some current 64bit OS have as a programming model.

Categories: CollectedKnowledge, HardWareCpu

Learn More

• http://www.amdboard.com/hammerspecial.html,
• Porting to AMD64: FAQ,
• Intel 64bits memory extension (ia32e, now known as EM64T)
• wikipedia definition

Glossary -- ProtectedMode

Protected mode is the 32 bit 'native' operating mode of Intel processors (and clones) since the 80386. It allows the developer to work with several virtual address spaces, each of which has 4GB of addressable memory and allows the system to enforce strict memory protection as well as restricting the available instruction set (so that your application cannot control the hard disk directly while the kernel can ;)

Protected mode unleashes the real power of your CPU, so you better get informed about it if you are considering writing an OS. However, it will prevent you from using virtually any of the BIOS interrupts (unless you have a V86 monitor).

Whether the CPU is in RealMode or in protected mode is defined by the lowest bit of the CR0 register, so basically

    ;; make sure interrupts are disabled, etc.
    mov eax, cr0
    or al,1
    mov cr0,eax

takes you to protected mode ... however you'll discover that there are many other things to be done before and after that operation to switch gracefully to pmode rather than resetting the CPU...

Plenty of information about protected mode can be found on both OSRC and Bona Fide, including in-detail tutorials and realmode/pmode switch programs. Our BabyStep series could also help you a bit :)

You may like http://home.swipnet.se/smaffy/asm/info/embedded_pmode.pdf if you're looking for a pragmatic tutorial on pmode.

Glossary -- RealMode 16 bits Operating mode in which the x86 cpu runs when it boots. That mode is mainly for backward compatibility and provide very few help for the modern developer (no memory protection, only 1MB of adressable memory, no virtual memory support). BIOS and DOS are typically real-mode stuff. All the rest you know (windows, linux, DukeNukem3D, zsnes, dos4gw, djgpp ...) are ProtectedMode OS/applications/dosextenders respectively.

additionnal informations about address formations in RealMode can be found in Perica's tutorial on Bona Fide.

When the PC boots up, the PIC IRQ's are mapped to interrupts 8 to 15 and 70 to 77.

So when your kernel boots up and you switch into protected mode, the PIC's remain unchanged, and the low 8 IRQ's end up corresponding to the same interrupt as CPU exceptions 8 to 15.

If one of these fire off, you can test if its an CPU exception or IRQ by testing some bits in the PIC status register but there is a much easier way, remap the PIC to use different interrupts than the CPU exceptions! See Can I remap the PIC? for more information on how this is done.

Categories: HardWareIrq

The PIC is a "Programmable Interrupt Controler" and is one of THE important chips (datasheets and more on OSRC), without it, x86 would not be an interrupt driven architecture.

There needs to be a way for perhiperals and other devices external of the CPU to tell the system than an event has happened or needs to happen. Examples of this: hard disk IO, modem/serial ports, keyboard.

Without the PIC interface, you would have to poll all the devices in the system to see if they want to do anything (signal an event), but with the PIC, your system can run along nicely until such time that a device wants to signal an event, which means you don't waste time going to the devices, you let the devices come to you when they are ready.

In the begining, the age of the IBM XT, we had only 1 PIC chip giving us 8 hardware interrupt lines, but the 8259A PIC chip has a neat ability, it can cascade!

Cascading means you can daisy chain PIC chips together. This is what happened with the introduction of the IBM AT, we had a second PIC chip cascaded onto the first, giving us a total of 15 hardware lines... Why 15 and not 16? That's because when you cascade chips, the PIC needs to use one of the int lines to signal to the other chip.

Thus, in an AT, IRQ line 2 is used to signal the second chip... But to confuse things more, IRQ 9 is redirected to IRQ 2. So when you get an IRQ 9, the signal is redirected to IRQ 2.

Categories: CollectedKnowledge, HardWareIrq related thread: edge/level-triggered interrupts

What's the PIC anyway ?

• P*rogrammable *I*nterrupt *C*ontroller. See InterruptsForDummies and What is the PIC? for details ...

There are two PIC "chips" (8259A) in PC design which are organized in a master/slave scheme: everything the slave controller has to report goes through a line of the master controller (hardwired on master's channel 2 in the PC design).

What's that noise about remapping ?

PICs can be configured to use a "vector offset" that is added to their IRQ line numbers to form interrupt vectors. Master and Slave PICs have each its own offset, independent of one another.

The default (BIOS-defined) vector offsets are 8 for Master PIC and 0x70 for Slave PIC:

• Master: IRQ 0..7 -> INT 8..0xF
• Slave: IRQ 8..15 -> INT 0x70..0x77

These default values don't suit the needs of ProtectedMode programming: there's a collision between IRQs 0..7 (mapped to INT 8..0xF) and processor exceptions (INT 0..0x1F are reserved).

from Intel manual v.3 p.5-2

"Vectors in the range 0 through 31 are reserved by the IA-32 architecture for architecture-defined exceptions and interrupts. Not all of the vectors in this range have a currently defined function. The unassigned vectors in this range are reserved. Do not use the reserved vectors.

The vectors in the range 32 to 255 are designated as user-defined interrupts and are not reserved by the IA-32 architecture. These interrupts are generally assigned to external I/O devices.. "

It's thus recommended to change the PIC's offsets (also known as remapping the PIC) so that IRQs use non-reserved vectors. A common choice is to move them to the beginning of the available range (IRQs 0..0xF -> INT 0x20..0x2F). For that, we need to set Master's offset to 0x20 and Slave's to 0x28.

This can be done by calling remap_pics(0x20, 0x28); (see code below).

Note however that,

• each PIC vector offset must be divisible by 8, as the 8259A uses the lower 3 bits for the interrupt number of a particular interrupt (0..7).
• the only way to change the vector offsets used by the PIC is to re-initialize it, which explains why the code is "so long" and plenty of things that have apparently no reasons to be here.
• if you plan to return to real mode (for any purpose), you really must restore the PIC to its former configration.

Programming the PIC chips

Each chip (master and slave) has a command port and a data port (given in the table below). When no command is issued, the data port allows us to access the interrupt mask of the PIC.

PIC ports:

A common command for the PIC is the end of interrupt command (code 0x20), but there's also the "initialize" command (code 0x11), which makes the PIC wait for 3 extra "initialization words" on the data port. Those data bytes gives the PIC

• its vector offset (ICW2),
• tell it how it is wired to master/slaves (ICW3)
• gives additionnal infos about the environment (ICW4)

Code

/* reinitialize the PIC controllers, giving them specified vector offsets
   rather than 8 and 70, as configured by default */

#define PIC1            0x20           /* IO base address for master PIC */
#define PIC2            0xA0           /* IO base address for slave PIC */
#define PIC1_COMMAND    PIC1
#define PIC1_DATA       (PIC1+1)
#define PIC2_COMMAND    PIC2
#define PIC2_DATA       (PIC2+1)
#define PIC_EOI         0x20            /* End - of - interrupt command code */

#define ICW1_ICW4       0x01            /* ICW4 (not) needed */
#define ICW1_SINGLE     0x02            /* Single (cascade) mode */
#define ICW1_INTERVAL4  0x04            /* Call address interval 4 (8) */
#define ICW1_LEVEL      0x08            /* Level triggered (edge) mode */
#define ICW1_INIT       0x10            /* Initialization - required! */

#define ICW4_8086       0x01            /* 8086/88 (MCS-80/85) mode */
#define ICW4_AUTO       0x02            /* Auto (normal) EOI */
#define ICW4_BUF_SLAVE  0x08            /* Buffered mode/slave */
#define ICW4_BUF_MASTER 0x0C            /* Buffered mode/master */
#define ICW4_SFNM       0x10            /* Special fully nested (not) */

/*
  arguments:
    offset1 - vector offset for master PIC
      vectors on the master become offset1..offset1+7
    offset2 - same for slave PIC: offset2..offset2+7
 */
void remap_pics(int offset1, int offset2)
{
        UCHAR   a1, a2;

        a1=inb(PIC1_DATA);   // save masks
        a2=inb(PIC2_DATA);

        outb(PIC1_COMMAND, ICW1_INIT+ICW1_ICW4);  // starts the initialization sequence
        io_wait();
        outb(PIC2_COMMAND, ICW1_INIT+ICW1_ICW4);
        io_wait();
        outb(PIC1_DATA, offset1);                    // define the PIC vectors
        io_wait();
        outb(PIC2_DATA, offset2);
        io_wait();
        outb(PIC1_DATA, 4);                       // continue initialization sequence
        io_wait();
        outb(PIC2_DATA, 2);
        io_wait();

        outb(PIC1_DATA, ICW4_8086);
        io_wait();
        outb(PIC2_DATA, ICW4_8086);
        io_wait();

        outb(PIC1_DATA, a1);   // restore saved masks.
        outb(PIC2_DATA, a2);
}

Q

What does that io_wait() function do ?

A

It forces the CPU to wait a little before going on, so that the PIC got the time to react. Simply jumping forward a few times or doing a small loop is usually enough. The exact timing doesn't really matter.

Note that even linux kernel is weird regarding to this feature, allowing a REAL_SLOW_IO flag make delays with 4 times more jumps or by writing to a 'dummy' port (0x80)

Q

Am i the only one to think ICW4_8086|ICW4_BUF_MASTER and ICW4_8086|ICW4_BUF_SLAVE should be sent to the PIC instead of raw 1 ?''

-- PypeClicker

A

Yes, you are. ;) Under normal circumstances, the PIC chip uses the SP/EN pin as an input pin to determine whether it is master or slave. Setting the BUF bit (3) in the PIC ICW4 causes the chip to instead use the SP/EN pin as an output pin for activating external buffers. Since PC-style computers are not wired up this way, that bit should never be set, although it probably doesn't do any harm other than requiring you to then use the M/S bit (2) to tell each PIC its function (since it's no longer using the SP/EN pin to tell). This information comes from The Indispensable PC Hardware Book by Hans-Peter Messmer.

-- DaidalosGuy

Further reading

• "The Indispensable PC Hardware Book" by Hans-Peter Messmer explains it nicely.

• The datasheet from Intel - "8259A Programmable Interrupt Controller"

  • Utterly fails to explain what is going on. Has pinout and electrical signal levels, in case you need them. It should have the words "Abandon hope.." written in large, burning letters on the cover..

Categories: HowTo, HardWareIrq

The NMI ("Non Maskable Interrupt") is a hardware-driven interrupt much like the PIC interrupts, but the NMI goes directly to the CPU, not via the PIC controller.

Luckily, you CAN have control over the NMI -- otherwise you could be in deep trouble.

The NMI is "turned on" (set high) by the memory module when a memory parity error occurs.

You have to be careful about disabling the NMI and the PIC for extended periods of time: your system will hang unless it has a failsafe timer! (You've always got one, as long as you don't kill the PIT timer.)

/* enable the NMI */
void NMI_enable(void)
{
        outb(0x70, inb(0x70)&0x7F);
}

/* disable the NMI */
void NMI_disable(void)
{
        outb(0x70, inb(0x70)|0x80);
}

Comments

Is it really wise to turn off NMI? okay, if you get an NMI while you're switching from RealMode to ProtectedMode, you could get a Triple Fault, which would reset the system, but isn't a system reset wished when content from memory is unreliable by that time ? -- Pype.

Is there much you can do if an NMI occurs? I guess if you got the error while reading from something that was copied from a disk at some point in the past and not modified since then you could read it from the disk again and continue with the (hopefully) good copy, but if it's something that has changed or been created dynamically then you don't really have an easy way to recover other than essentially starting from scratch. -- Kemp.

Of course, you're assuming that the kernel code hasn't been corrupted, if the kernel is damaged then you'll most likely tripple fault anyway. The best course of action is probably to request the user perform a RAM diagnostic with MemTest or something (and hope you don't crash before you can get that far). Then again... Windows keeps a "Hardware Damaged" flag for every physical page of RAM, unfortunately this would mean aborting the program that was running when the NMI occured then checking each page the program was using at the time for what are basically "bad sectors" and flaging them so they aren't used again -- AR

So if an NMI occurs you should assume it'll continue happening at that location in future rather than it being a freak occurrence? -- Kemp

I'm not an engineer so I wouldn't know for certain, but I would be inclined to think that if an NMI occurs on a page when you retest the program that caused the original NMI then it would seem rather likely that the chip is faulty, I don't know if Windows persists the flags across reboots but I doubt it since AFAIK it's impossible to tell if the RAM has been replaced while the computer was off. If the NMI doesn't occur again while checking the pages then the fault isn't severe and could be written off, just if it is a persistent problem then the page should probably be disabled (in the interest of preventing random program crashes) -- AR

Categories: CollectedKnowledge, HardWareIrq

What is an Interrupt Service Routine (ISR)?

The x86 architecture is an interrupt driven system. External events trigger an interrupt - the normal control flow is interrupted and a interrupt service routine (ISR) is called.

Such events can be triggered by hardware or software. An example of a hardware interrupt is the keyboard: Every time you press a key, the keyboard triggers IRQ1 (Interrupt Request 1), and the corresponding interrupt handler is called. Timers, and disk request completion are other possible sources of hardware interrupts.

Software driven interrupts are triggered by the int opcode; e.g. the services provided by MS-DOS are called by the software triggering INT 21h and passing the applicable parameters in CPU registers.

For the system to know which interrupt service routine to call when a certain interrupt occurs, offsets to the ISR's are stored in the interrupt descriptor table (IDT) when you're in ProtectedMode, or in the interrupt vector table (IVT) when you're in RealMode.

More about ISR, IRQ and stuff can be found on Bona Fide.

What makes an ISR so special?

An ISR is called directly by the CPU, and the protocol for calling an ISR differs from calling e.g. a C function. Most importantly, an ISR has to end with the iret opcode, whereas usual C functions end with ret or retf. The obvious but nevertheless wrong solution leads to one of the most "popular" tripple-fault errors among OS programmers.

The Problem

Many people shun away from Assembler, and want to do as much as possible in their favourite high-level language. GCC (as well as other compilers) allow you to add inline Assembler, so many programmers are tempted to write an ISR like this:

/* How NOT to write an interrupt handler           */
void interrupt_handler(void)
{
    __asm__("pushad"); /* Save registers.          */
    /* do something */
    __asm__("popad");  /* Restore registers.       */
    __asm__("iret");   /* This will triple-fault! */
}

This cannot work. The compiler adds stack handling code before and after your function, which together with the iret results in Assembler code resembling this:

push   %ebp
mov    %esp,%ebp
sub    $<size of local variables>,%esp
pushad
# C code comes here
popad
iret
# 'leave' if you use local variables, 'pop %ebp' otherwise.
leave
ret

It should be obvious how this messes up the stack (ebp gets push'ed but never pop'ed). Don't do this. Instead, these are your options.

Solutions

Plain Assembler

Learn enough about Assembler to write your interrupt handlers in it. ;-)

Two-Stage Assembler Wrapping

Write an Assembler wrapper calling the C function to do the real work, and then doing the iret.

/* filename : isr_wrapper.asm */
.globl   _isr_wrapper
.align   4

_isr_wrapper:

    pushad
    call    _interrupt_handler
    popad
    iret


/* filename : interrupt_handler.c */
void interrupt_handler(void)
{
    /* do something */
}

Compiler Specific Directives

Some compilers for some processors have directives allowing you to declare a routine interrupt, offer a #pragma interrupt, or a dedicated macro. Borland C, Watcom C/C++, Microsoft C 6.0 and Free Pascal Compiler 1.9.* and up offer this, while VisualC++ and GCC don't:

/* Borland C */
void interrupt interrupt_handler(void)
{
    /* do something */
}

/* Watcom C/C++ */
void _interrupt interrupt_handler(void)
{
    /* do something */
}

Actually, VisualC++ can be used to make interupts, by making them naked. adding _declspec(naked) to your function will cause the compiler to leave out the stack handling code. This means you are free to set up and release stack space however you want. Just be careful that you put in a return statement, because the compiler won't, it will just allow execution to continue past the end of your code on into garbage. Also, if you plan to use local variables or function arguments in the C code, you need to set up the stack frame the way the compiler expects it. This is not such a problem in interrupts though, because since they are non-reenterant, you can simply use static variables.

/* Microsoft Visual C++ */
void _declspec(naked) interrupt_handler()
{
    _asm pushad;

    /* do something */

    _asm{
        popad
        iretd
    }
}

Black Magic

Look at the faulty code above, where the proper C function exit code was skipped, screwing up the stack. Now, consider this code snippet, where the exit code is added manually:

/* BLACK MAGIC - Strongly Discouraged! */
void interrupt_handler() {
    __asm__("pushad");
    /* do something */
    __asm__("popad; leave; iret"); /* BLACK MAGIC! */
}

The corresponding output would look somewhat like this:

push   %ebp
mov    %esp,%ebp
sub    $<size of local variables>,%esp
pushad
# C code comes here
popad
leave
iret
leave # dead code
ret   # dead code

This assumes that leave is the correct end-of-function handling - you are doing the function return code "by hand", and leave the compiler-generated handling as "dead code". Needless to say, such assumptions on compiler internals are dangerous. This code can break on a different compiler, or even a different version of the same compiler. It is therefore strongly discouraged, and listed only for completeness.

Categories: FAQ, HardWareIrq

See Also Help!? I can't get interrupts working

Related Threads

brendan providing a great intro on IRQ, ISR and similar stuff how to know the interrupt number in a generic handler

What's the APIC ?

APIC (Advanced Programmable Interrupt Controller) is the Intel standard for the "new" PIC. Its used in multiprocessor systems and is an integral part of all Intel (and compatible) processors from the P6 (Pentium III) and onwards.

The Intel APIC specification is used in modern processors, and there's a bit (bit 9) in the CPUID standard information flags that let you see whether a CPU has an APIC. The APIC itself is used for more up to date interrupt redirection, and for sending interrupts between processors. These things weren't possible using the older PIC specification.

Local APIC and IO-APIC

In APIC-based system, each CPU is made of a core and a local APIC. The local apic is responsible for handling cpu-specific interrupt configuration. Among other things, it contains the Local Vector Table (LVT) that translate events such as "internal clock" and other "local" interrupt sources into a interrupt vector (e.g. LocalINT1 pin could be raising an NMI exception by storing "2" in the corresponding entry of the LVT).

information about the local APIC can be found in "system programming guide" of intel processors)

In addition, the I/O APIC (e.g. intel 82093AA) is part of the chipset and provides multi-processor interrupt management, incorporating both static and dynamic symmetric interrupt distribution across all processors. In systems with multiple I/O subsystems, each subsystem can have its own set of interrupts.

Each interrupt pin is individually programmable as either edge or level triggered. The interrupt vector and interrupt steering information can be specified per interrupt. An indirect register accessing scheme optimizes the memory space needed to access the I/O APIC's internal registers. To increase system flexibility when assigning memory space usage, the I/O APIC's two-register memory space is re-locatable, but defaults to 0xFEC00000.

The original I/O APIC specification/datasheet is available from Intel at http://www.intel.com/design/chipsets/datashts/290566.htm and an updated version can be found at http://developer.intel.com/design/chipsets/specupdt/290710.htm.

The Intel Standards for the APIC can be found on the Intel site under the name of Multiprocessor Specification at http://developer.intel.com/design/pentium/datashts/24201606.pdf.

Inter-Procesor Interrupts

Inter-Processor Interrupts (IPIs) are generated by local APIC and can be used as basic signalling for scheduling coordination, multi-processors bootstrapping, etc.

APIC configuration

The local APIC is enabled at boot-time and can be disabled by clearing bit 11 of IA32_APIC_BASE_MSR (the CPU then receives its interrupts directly from a 8259-compatible PIC, which is usually used prior a reboot) and the I/O APIC can be programmed in legacy mode (so that it reacts as a 8259 device).

The local APIC registers are memory-mapped in physical page FEE00xxx (as seen in table 8-1 of intel P4 SPG). Note that there's a ModelSpecificRegister that specifies the actual APIC base.

Make sure you enable the APIC, by OR-ing the Spurious Interrupt Register (0x00F0) with 0x100, before you try to configure anything else ;)

#define IA32_APIC_BASE_MSR 0x1B
#define IA32_APIC_BASE_MSR_ENABLE 0x800

/** returns a 'true' value if the CPU supports APIC
 *  and if the local APIC hasn't been disabled in MSRs
 *  note that this requires CPUID to be supported.
 */
boolean cpuHasAPIC()
{
   dword a,d;
   cpuid(1,&a,&d);
   return d&CPUID_FLAG_APIC;
}

/** defines the physical address for local APIC registers
 */
void cpuSetAPICBase(phys_addr apic)
{
   dword a=(apic&0xfffff000) | IA32_APIC_BASE_MSR_ENABLE;
#ifdef __PHYSICAL_MEMORY_EXTENSION__
   dword d=(apic>>32) & 0x0f;
#else
   dword d=0;
#endif

   cpuSetMSR(IA32_APIC_BASE_MSR, a,d);
}

/** determines the physical address of the APIC registers page
 *  make sure you map it to virtual memory ;)
 */
phys_addr cpuGetAPICBase()
{
   dword a,d;
   cpuGetMSR(IA32_APIC_BASE_MSR,&a,&d);
#ifdef __PHYSICAL_MEMORY_EXTENSION__
   return (a&0xfffff000)|((d&0x0f)<<32);
#else
   return (a&0xfffff000);
#endif
}

IO APIC Configuration

The IO APIC, like the VGA controller and the CMOS, has a two-register address space - an address register at IOAPICBASE+0 and a data register at IOAPICBASE+0x10. All accesses must be done on dword boundaries. Here's some example code that illustrates this:

dword cpuReadIoApic(void *ioapicaddr, dword reg)
{
   dword * volatile ioapic = (dword*)ioapicaddr;
   ioapic[0] = reg;
   return ioapic[4];
}

void cpuWriteIoApic(void *ioapicaddr, dword reg, dword value)
{
   dword * volatile ioapic = (dword*)ioapicaddr;
   ioapic[0] = reg;
   ioapic[4] = value;
}

Note the strange usage of the volatile keyword. It means in this case that the value pointed to, not the pointer, is volatile. It prevents the compiler, as Visual C does, from reordering the memory accesses, which is a Bad Thing.

Categories: CollectedKnowledge, HardWareIrq

Related threads

APIC timer "mapping IO APIC"

Disclaimer: Hi, this is PypeClicker typing. I'm using this page as a scratchpad for a yet-to-come USB programming tutorial.

What happens on a USB cable ?

USB looks much more like a network protocol than like RS232 (good old serial cables). Data transmissions follows formatted packets rather than being made of plain characters. However, unlike network protocols like Ethernet, Token Ring, etc. all communications are directed by the host (i.e. your computer), and even 'interrupts' are actually polled by the host.

More informations about this section can be found on Usb In A Nutshell (ch. 3)

USB Packets

These are the smallest formatted things that may transfer on the USB cable. Each USB packet at least have a SYNC header and a EOP trailer that carry no information but help identifying packets boundaries. Each USB must also have a PID field (Packet IDentifier) that tells the role of the packet in a transfer or a transaction.

USB transaction

A transaction is a "single sentence" between the host and a device. Transactions may be used to send/read data from the device, initiate transfers, set up parameters, get information, etc. Each transaction is typically made of 3 packets:

1. a token packet that always comes from the host and carries the address of the device/endpoint concerned in the transaction. USB defines 4 types of tokens: SETUP (initiates a control transaction), IN (initiates a device->host data transaction) and OUT (initiates a host->device data transaction). Tokens are typically have small fixed size.
2. a data packet which carries the transaction payload. Depending on the USB version, transfer speed and type of data packet used (DATA0, DATA1, DATA2 or MDATA), the maximum payload may be of 8, 64 or 1024.
3. a handshake packet which informs the data emitter of the reception status. Handshake may be ACK (data received correctly), NAK (unable to receive/emit data right now, or no data to send) or STALL (device state invalid, host intervention required)

USB transfers

Transactions are used to build more complex protocols like control transfer, interrupt transfer (single small sized status poll), isochronous transfer (periodic non-acknowledged packet transmission, like a sample to play on speakers) and bulk transfer (non-predictable large-sized transfers like a page to print)

More on USB transfers can be found on USB in a nutshell, ch 4

further reading required to know how much on transfers will be required by the implementor

USB Configuration

The configuration part of the USB standard occurs through the notion of descriptors. Descriptors are blocks of information retrieved through the host controller which defines things like vendor/product identifiers for each device and class/subclass/protocol codes for each device's interface. Based on these informations, the Operating System can assign the proper driver to each device/interface.

A list of structures defining USB descriptors is available on Clicker's CVS (usbdescr.h)

note that Clicker is (L)GPL. If someone finds some equivalent text in public domain, feel free to post it here

Endpoints

Configuration communications are performed on endpoint 0. The USB device will usually define more endpoints used as communication channels between the device and the host, link those endpoints to interfaces which can be assigned a class, subclass and protocol codes (identifying what the device is able to do). For instance a digital camera could offer a couple of endpoints to implement the USBstorage interface (for the camera's memory card) and other endpoints for a webcam interface.

Reading descriptors

The host has to explicitly request descriptors at the device before it manipulate them. This is achieved by a setup transfer using a GET_DESCRIPTOR as the host->device data and receiving the descriptor as the data from the host. The whole transfer will look like

1. command transaction:

    Setup_TOKEN(addr=device, endpoint=0) : host -> device
    DATA[ RequestType=0x80, Request=0x06, value=DESCR_TYPE|DESCR_SELECTOR,
           index=0, length=wished_length] : h->d, 8 bytes
    ACK : host <- device

1. response transaction:

    IN_TOKEN(addr=device, endpoint=0) : host -> device
    DATA(the_descriptor) : host <- device (as much bytes as requested)
    ACK : host -> device

1. confirm transaction:

    OUT_TOKEN(addr=device, endpoint=0) : host -> device
    DATA() : host -> device (empty)
    ACK : host <- device

The DATA packet in the command transaction is called "Setup Packet" (according to Beyond Logic), and carries almost all of the 'interresting' stuff:

• the RequestType of 0x80 (DeviceToHost=0x80| StandardRequest=0| DeviceTargetted=0)
• the Request (command) 0x06 for "GET_DESCRIPTOR"
• the value (encoding unknown atm)

categories: HardWareBus