[NTLK] Newt/64 and Newton Script internals

[Matthias Melcher]

Even though this is the usual extreme geekyness you expect from me, here are some gory details on newt/64 and NewtonScript packages.


So, newt/0 is the awesome NewtonScript compiler and interpreter by Makoto Nukui, aka gnue. I ported newt/0 to run an 64bit machines (hence newt/64), and I added functionality to read and write .pkg files that contain NewtonScript apps. In the last weeks, I have tried to make the process so transparent, that newt/64 can read any package, create the original data structure, print everything into a text file, so we can see the data and the code of the original app.

We can then edit the code, fix bugs, update it, whatever, and then simply recompile it back into a .pkg file that should run and behave just like the original file.

In code, that looks a bit like that:


printDepth := 9999;
printLength := 9999;
printBinaries := 1;
printUnique := 1;

bin := LoadBinary( "some.pkg" );
app := ReadPkg( bin );
p(app);


The output should be a compileable version of the original app, so then we take the output and do:


pkg := { ... output from the code above ... };
bin := MakePkg(pkg);
SaveBinary(bin, "someNew.pkg");


...and we should have functionally the same app as before (possibly including fixes).


This is the first milestone. I have tried this and it works. It may need a few lines of additional code, but the basics are there. It doen *not* yet reverse the bytecode, but that is absolutely possible, as ViewFrame has shown. If I can implement that, then we can reverse engineer almost any package there is. Milestone 2.

This is the basis for the next huge milestone then is to run packages in newt/64 and implement the graphics calls that are needed. This would make packages run in a native interpreter at *full speed* on the host machine *without* the need of a ROM, at least for some chosen apps. Hehehe.

 - Matthias

---

I have been playing around a little bit with newt/64 and I have been trying to fix the pkg import and export, when I found some rather interesting details. I fixed a few bugs and basically, we can now read any NSOF package, look at the internal structures and print them, possibly change them, and then create a package again from the code.

There are still some details that I want to solve. This should give us a way to fix any existing package:
	• this is not reverse engineering byte code, but since ViewFrame has solved this, I am hoping to be able to back port that code
	• some arrays (that my later become maps when running code) must have their class set to an integer, this is possible, but not yet automated
	• new functions have a class that can't be represented in NewtonScript, so I added the syntax "class: $\s0003" to allow that
	• most binaries use MakeBinaryFromHex to create them; for images and sounds, we should create resource files, so they can be edited, and link them back in at compile time
	• you can set printUnique:=1; which will tell the print routine to print objects only the first time they are referenced, and then just print a rf to them
	• we could make packages smaller if we throw out duplicate maps and just reference them multiple times instead

https://github.com/MatthiasWM/NEWT64 , run 'newt' and check out the 'newt.ns' file to get an idea of what I am trying to do.