This reminds me of some playing around I did with x86 instruction decoding. I've sort of come to a conclusion that both the assembly and the manual are lying to you a little bit about the actual instruction formats.
The basic form of an x86 instruction is as follows:
* The presence or absence of each of 5 "legacy" prefixes (0x66, 0x67, 0xf0, 0xf2, 0xf3). Supposedly, 0xf0, 0xf2, and 0xf3 can't coexist (they're all group 1), but if you look carefully at a few instructions, it turns out that there are some instructions which require you to specify two of them at the same time.
* The group 2 prefixes end up being a segment register specifier, although they got overloaded as branch hits for the branch instructions.
* If you use a REX prefix, it adds an extra bit to the opcode (REX.w). If you use a VEX prefix, it adds two extra bits (REX.w and VEX.l). EVEX adds five extra bits (REX.w, VEX.l + EVEX.l, EVEX.b, EVEX.z). Otherwise, everything they add is extra register bits, although some instructions may have differences if they use a "modern" prefix versus not.
* Opcode map. For VEX and EVEX, you specify this via a few bits in the prefix, but otherwise, you use 0x0f, 0x0f 0x38, or 0x0f 0x3a to specify.
* 1-byte opcode.
Combine all of these together, and you almost have every unique instruction. That's roughly ~21 bits of opcode encoding (fewer due to holes and impossible encodings), and this will tell you what the instruction expects as its opcode form: whether it has a ModR/M byte and how many subsequent immediate bytes it expects. The ModR/M byte specifies a register and either a register or a memory location. It actually doesn't take that long to iterate through every single x86 instruction!
However, there's one last trick x86 plays: sometimes, the register of the ModR/M byte is used to select between different instructions, particularly when you have an instruction that only takes memory instructions. So the instruction 0x0f 0x01 with r0 is a SGDT instruction, whereas the instruction 0x0f 0x01 with r3 is a LIDT instruction instead.
Wow that's a very interesting summary of the encoding of opcode itself. Thanks!
I find this part the most challenging. It is relatively easy to figure out the "mapping between the assembly and instruction" when you have both in front of you already, as I did in my posts.
Note for other people reading both my post and the comment above:
The terms r0 and r3 in the last paragraph corresponds to what I describe in my post as: The "register code" or "opcode extension" values stored in the "ModR/M.reg" field. In this case, the values 0 and 3 are meant to be "opcode extensions". You can see both instructions here: http://ref.x86asm.net/coder64.html#x0F01 . The values 0 (for SGDT) and 3 (for LIDT) are shown in the column called "o", which is defined as: "Register/ Opcode Field"
* F0 and F2 are not mutually exclusive; there are few instructions that use both prefixes simultaneously.
* It's a little bit more helpful, IMHO, to think of the prefixes not as prefixes but as extra bits to the opcode, so 0x66 0x00 and 0x00 are different instructions that just happen to both be ADD of different sizes.
In general, the section on "legacy" prefixes seems to have largely been untouched since largely the 32-bit days, and so it's generally written as if it's mostly talking about 8086-style instructions whereas the actual implementation (especially when you start getting to the SSE instructions) has diverged somewhat from the original meanings of those prefixes, instead just stealing them for extra opcode bits.
The basic form of an x86 instruction is as follows:
* The presence or absence of each of 5 "legacy" prefixes (0x66, 0x67, 0xf0, 0xf2, 0xf3). Supposedly, 0xf0, 0xf2, and 0xf3 can't coexist (they're all group 1), but if you look carefully at a few instructions, it turns out that there are some instructions which require you to specify two of them at the same time.
* The group 2 prefixes end up being a segment register specifier, although they got overloaded as branch hits for the branch instructions.
* If you use a REX prefix, it adds an extra bit to the opcode (REX.w). If you use a VEX prefix, it adds two extra bits (REX.w and VEX.l). EVEX adds five extra bits (REX.w, VEX.l + EVEX.l, EVEX.b, EVEX.z). Otherwise, everything they add is extra register bits, although some instructions may have differences if they use a "modern" prefix versus not.
* Opcode map. For VEX and EVEX, you specify this via a few bits in the prefix, but otherwise, you use 0x0f, 0x0f 0x38, or 0x0f 0x3a to specify.
* 1-byte opcode.
Combine all of these together, and you almost have every unique instruction. That's roughly ~21 bits of opcode encoding (fewer due to holes and impossible encodings), and this will tell you what the instruction expects as its opcode form: whether it has a ModR/M byte and how many subsequent immediate bytes it expects. The ModR/M byte specifies a register and either a register or a memory location. It actually doesn't take that long to iterate through every single x86 instruction!
However, there's one last trick x86 plays: sometimes, the register of the ModR/M byte is used to select between different instructions, particularly when you have an instruction that only takes memory instructions. So the instruction 0x0f 0x01 with r0 is a SGDT instruction, whereas the instruction 0x0f 0x01 with r3 is a LIDT instruction instead.