The instruction set is part of the VM/CPU, however I guess you
could make a custom lanuage that compiles down to 6502 machine
code, which is what Hacker's Edge runs.
Most of the initial commands will use the initial KERNEL.SYS API,
this API includes writing a stringM input, file IO, networking,
and other functions. if you wrote your own kernel, then the API
itself could be entirely different, which would result in your
binaries only running on hosts that have your custom kernel.
There is a game guide on the current binary format, and you should
be able to eventually write your own format, yes. the executable
machine code will still need to be 6502, but the header, and how
it loads into memory can be entirely custom.