Java Primer
This is a demonstration tutorial of how you could write an OS using Java. The methods described in here however are quite generic - you can use similar approaches for other bytecoded languages, such as .NET. In addition, this article contains a simple compiler that can be a source of inspiration for any custom languages.
Being a demonstration, the method posted here is exceedingly simple and only does the necessary work to make the bundled Java code run. It can not deal with most things one would expect from Java. It does not do garbage collection in any forms, let alone support strings. The code provided here is purposefully limited, so you can play with it a bit to get a feeling, but it is impossible to extend this code to support more key components of the language.
Consider this a throwaway prototype - it's written in such a fashion that you're forced to rewrite a new one from scratch.
Supporting a non-native language
Since you have much less to leech from existing compilers, you essentially are in charge of the entire toolchain. There are a number of rough steps:
- Define your native ABI
- Java and .NET come in bytecode, which are designed for machines that are fundamentally different than your typical register-based hardware. You'll need to understand assembly for your platform, and you'll have to think about how the language structures should be converted to native ones. One of the things this example does is to adhere to the stdcall convention, because it's closer to the virtual java machine than the cdecl that's the default in most compilers.
- Create bytecode-to-native compiler
- Java comes in bytecode form, and we need a method of converting it to native. Because we are writing an OS at the same time, everything must be compiled ahead of time, and not rely on an existing runtime that expects the existing presence of an OS. Therefore, we write our own compiler. As part of the compiler, we include the org.objectweb.asm that can read .class files and save us from writing significant portions of code.
- Compile the compiler
- The compiler is of course written in Java as well. We don't need additional languages to add to the bootstrap problem later. Of course this compiler is overly simplistic, and will never be able to compile itself.
- Compile managed os to bytecode
- Javac is written in Java, so that's obviously the best choice for later porting.
- Compile bytecode to native assembly
- Now that we have the compiler, we can run it on all our OS classes to generate native code.
- Create runtime for things that have to be non-native
- Java is memory-safe, which prevents it from implementing a lot of things. A minimal set of startup code is done in native assembly. Everything architecture-specific has to be done in assembly. Some parts you'll need later, such as memory management and exception handling are probably easier to have pure native parts for as well.
- Assemble os and runtime
- We have everything in machine language form now. We're still reuseing the parts that we can, and in this example we use yasm. You're free to write an assembler in Java later on.
- package the final kernel binary
- This is done using binutils. Note that there is not a single C compiler in the chain, and again, you can build your own linker in your language - it doesn't really have to do that much.
Compiler
In bytecoded languages there are several steps before code can be run. Typically you have "the" compiler, which converts your source files into some portable binary, and you have an interpreter that reads those binaries and runs instructions from them. Modern interpreters turn bytecode into native code, as to avoid the if(instruction = ...) that takes several cycles while the instruction you'd actually want to execute would otherwise cost you just one CPU instruction.
In the case of an OS, we need to take this a step further. We could run an interpreter, but that's slow. We could compile into native on boot but that needs just as much OS as we actually want to run. Instead, the appropriate solution is to compile to native in advance, so we can just run the code directly from the start.
The entirety of a language is still a whole lot to deal with, but for our example it suffices to deal with just integers. That's right, no objects yet!
The Java bytecode uses a stack for operations, and a list of locals. These need not be in the same place, but as the x86 only has one hardware stack, we'll be using it as both local stack, call stack, and operation stack. A few tricks are used to make this compiler easier, and in turn, make it difficult to interface with C. Locals in Java terms include the function arguments, and as a result the locals would be split around return addresses. Since the caller doesn't know the storage needed - it doesn't even get the number of arguments for free - the called function should fix this. We also can't put things past the top of the stack because that'll be a big issue with interrupts later. Basically we copy all arguments to the other side of the return address and then we make some room for locals so that they can be indexed by EBP - 4 * slot_number where for instance 0 and 1 would be arguments and 2+ would be true locals.
The fact that getting the number of arguments is convoluted to perform on the caller's side, we do callee-cleanup using RET imm instead of the regular RET. Locals is a convoluted issue as well, so we just reserve room for 8 because you're not meant to copy this code anyway.
package nl.combuster.minijava;
import org.objectweb.asm.*;
import org.objectweb.asm.tree.*;
import java.util.*;
import java.io.*;
public class Compiler
{
public static byte[] readEntireFile(String filename)
{
File file = new File(filename);
try
{
FileInputStream input = new FileInputStream(file);
byte bytes[] = new byte[(int)file.length()];
input.read(bytes);
return bytes;
}
catch (IOException e)
{
throw new RuntimeException("Unable to read file " + file, e);
}
}
public static void writeOutput(String filename, List<String> lines)
{
try
{
BufferedWriter writer = new BufferedWriter(new FileWriter(new File(filename)));
for (String string : lines)
{
writer.write(string);
writer.newLine();
}
writer.close();
}
catch (IOException e)
{
throw new RuntimeException("Unable to write output file " + filename, e);
}
}
public static String decorate(String classname, String method, String signature)
{
// make all these assembly-friendly names. Note that the
// constructor is for instance called <init>
return classname.replace("/","_") + "__" + method.replace("<","_").replace(">","_");
}
public static void jumpgroup(List<String> output, String jumpcode, LabelNode dest)
{
output.add("pop edx");
output.add("pop ecx");
output.add("cmp ecx, edx");
output.add(jumpcode + " .l" + dest.getLabel());
}
public static void main(String args[])
{
if (args.length != 2) throw new RuntimeException("Usage: compiler input-file output-file");
ClassNode node = new ClassNode();
ClassReader reader = new ClassReader(readEntireFile(args[0]));
reader.accept(node, 0);
List<String> outputdata = new LinkedList<String>();
outputdata.add("section .text");
for (MethodNode method : node.methods)
{
method.visitCode();
String methodname = decorate(node.name, method.name, method.signature);
if ((method.access & Opcodes.ACC_NATIVE) != 0) continue;
// prologue
System.out.println("; attributes: " + method.attrs);
outputdata.add("global " + methodname);
outputdata.add(methodname + ":");
outputdata.add("push ebp");
outputdata.add("mov ebp, esp");
int locals = (method.localVariables == null) ? 0 : method.localVariables.size();
outputdata.add("; locals: + " + locals);
int arguments = (method.parameters == null) ? 0 : method.parameters.size();
if ((method.access & Opcodes.ACC_STATIC) == 0) arguments++; // hidden "this"
outputdata.add("; params: + " + arguments);
// copy params so that they correspond with java indexing and join with the local numbering
for (int i = 0; i < arguments; i++)
{
outputdata.add("push dword [ebp + " + (8 + 4 * i) + "]");
}
// do some frame checking for many variables, locals is not of much use...
outputdata.add("sub esp, 32");
Iterator<AbstractInsnNode> iterator = method.instructions.iterator();
while (iterator.hasNext())
{
AbstractInsnNode insn = iterator.next();
int opcode = insn.getOpcode() & 0xff;
outputdata.add(" ; " + opcode + " = " + insn.getClass().getSimpleName());
//outputdata.add("mov byte [" + (0xb8000 + 156) + "], '0' + " + (opcode % 10));
//outputdata.add("mov byte [" + (0xb8000 + 154) + "], '0' + " + (opcode / 10)%10);
//outputdata.add("mov byte [" + (0xb8000 + 152) + "], '0' + " + (opcode / 100));
if (insn instanceof LabelNode)
{
LabelNode labelinsn = (LabelNode)insn;
outputdata.add(".l" + labelinsn.getLabel());
}
else if (insn instanceof LineNumberNode)
{
// ignore these
}
else if (insn instanceof VarInsnNode)
{
// copy a variable
VarInsnNode varinsn = (VarInsnNode) insn;
switch(varinsn.getOpcode())
{
case Opcodes.ILOAD:
case Opcodes.ALOAD:
// todo: verify offset
outputdata.add("push dword [ebp - " + (4 + 4 * varinsn.var) + "]");
break;
case Opcodes.ISTORE:
case Opcodes.ASTORE:
outputdata.add("pop dword [ebp - " + (4 + 4 * varinsn.var) + "]");
break;
default:
outputdata.add("Can't deal with varinsnnode: " + varinsn.getOpcode());
}
}
else if (insn instanceof MethodInsnNode)
{
MethodInsnNode methodinsn = (MethodInsnNode) insn;
String calledmethod = decorate(methodinsn.owner, methodinsn.name, methodinsn.desc);
outputdata.add("extern " + calledmethod);
outputdata.add("call " + calledmethod);
if (!methodinsn.desc.endsWith("V"))
{
// not a void return value
outputdata.add("push eax");
}
/*switch (methodinsn.getOpcode())
{
default:
outputdata.add("Can't deal with methodcall: " + methodinsn.getOpcode());
}*/
}
else if (insn instanceof IntInsnNode)
{
IntInsnNode intinsn = (IntInsnNode)insn;
switch(intinsn.getOpcode())
{
case Opcodes.BIPUSH:
outputdata.add("push " + intinsn.operand);
break;
default:
outputdata.add("Can't deal with intinsnnode: " + intinsn.getOpcode());
}
}
else if (insn instanceof JumpInsnNode)
{
JumpInsnNode jmpinsn = (JumpInsnNode)insn;
switch(jmpinsn.getOpcode())
{
case Opcodes.IF_ICMPEQ: jumpgroup(outputdata, "je", jmpinsn.label); break; // 159
case Opcodes.IF_ICMPNE: jumpgroup(outputdata, "jne", jmpinsn.label); break; // 160
case Opcodes.IF_ICMPLT: jumpgroup(outputdata, "jb", jmpinsn.label); break; // 161
case Opcodes.IF_ICMPGE: jumpgroup(outputdata, "jae", jmpinsn.label); break; // 162
case Opcodes.IF_ICMPGT: jumpgroup(outputdata, "ja", jmpinsn.label); break; // 163
case Opcodes.IF_ICMPLE: jumpgroup(outputdata, "jbe", jmpinsn.label); break; // 164
case Opcodes.GOTO: // 167
outputdata.add("jmp .l" + jmpinsn.label.getLabel());
break;
default:
outputdata.add("Can't deal with jumpinsnnode: " + jmpinsn.getOpcode());
}
}
else if (insn instanceof LdcInsnNode)
{
LdcInsnNode ldcinsn = (LdcInsnNode) insn;
if (ldcinsn.cst instanceof Integer)
{
outputdata.add("push " + ldcinsn.cst.toString());
}
else
{
outputdata.add("Can't deal with data in ldcinsnnode (" + ldcinsn.getOpcode() +"): " + ldcinsn.cst.getClass().getSimpleName());
}
}
else if (insn instanceof IincInsnNode)
{
IincInsnNode incinsn = (IincInsnNode) insn;
switch (incinsn.getOpcode())
{
case Opcodes.IINC:
outputdata.add("add dword [ebp - " + (4 + 4 * incinsn.var) + "], " + incinsn.incr);
break;
default:
outputdata.add("Can't deal with iincinsnnode: " + incinsn.getOpcode());
}
}
else if (insn.getOpcode() >= Opcodes.ICONST_M1 && insn.getOpcode() <= Opcodes.ICONST_5) // 2...8
{
outputdata.add("push " + (insn.getOpcode() - Opcodes.ICONST_M1 - 1));
}
else if (insn.getOpcode() == Opcodes.IADD) // 96
{
outputdata.add("pop edx");
outputdata.add("add [esp], edx");
}
else if (insn.getOpcode() == Opcodes.IMUL) // 104
{
outputdata.add("pop eax");
outputdata.add("pop ecx");
outputdata.add("imul ecx"); // eax:edx = eax * ecx
outputdata.add("push eax");
}
else if (insn.getOpcode() == Opcodes.I2B) // 145
{
outputdata.add("and dword [esp], 0xff");
}
else if (insn.getOpcode() == Opcodes.RETURN) // 177
{
outputdata.add("xor eax, eax");
}
else if (insn instanceof FrameNode)
{
outputdata.add("; framenode");
}
else
{
outputdata.add("Can't deal with " + insn.getClass().getSimpleName() + ", fix it (" + insn.getOpcode() + ")");
}
}
// epilogue, stdcall to save complexity on decoding methodinsns
outputdata.add("leave");
outputdata.add("ret " + arguments);
}
writeOutput(args[1], outputdata);
}
}
Yes, that's a compiler in a magic 256 lines of code. The result is Intel syntax assembly, because there already are decent tools for assembly out there that deal with object formats out there. Of course you can write your own later as well.