[NTLK] Einstein Xcode project cleanup

[Matthias Melcher]

Ooops, so sorry I missed the traffic on this one! I was very busy in my job the last weeks. I did have some time to play with cjit and improve it. By now, I can translate about 1700 functions.

I'll try to answer everything that came up in the previous mails:

1: the EinsteinDev target is the one that links the patches for cjit generated code. Plain "Einstein" has no patches beyond the original required ones. "EInsteinDev" is meant for us developers and should not be used as a release version.

2: GIT etiquette. None that I know of. I was always as careful as possible when changing stuff around, eventually getting bolder. The important thing should be that the code always compiles and runs without changes. For OS X, I always assume the newest Xcode version, simply because Xcode is free, so every dev should have it. If you improve the code or - equally important - improve the IDE scripts, and you are confident that it doesn't break anything, commit it to the main repo.

3: The network card emulation does not use the FFI library. It uses a driver package written in NewtonScript and ARM code that calls native code the same way the Pauls REx calls Einstein code.

4: "NewtSim" and "retarget". The NewtSim target was removed in the next commit. Instead, I added the command line tool "retarget". This little app runs a very limited set of "Einstein Monitor Scripts", basically "rt open", "rt cjit(r)", and "rt close" to programmatically create simulator code. "NewtonDev" now has a directory filled with scripts, automatically generating simulator code at compile time.

5: 64 bit: I have never tried that, but I admit that I have no paid any attention to beeing 64 bit compliant. It is therothically possible, that native pointers (64 bit) are stored somewhere in emulated code (32 bit), which would fail. There is alos a union that holds a value or a JIT function pointer. It should work, but I have not verified that.

6: on the "contribute" page, I had not gotten any direct offers, but I also have not looked at that page for a while either. It may be outdated.

7: MMU. The cjit stuff converts emulated code into more or less the same code that the JIT compiler would have generated, minus the conversion time (since that is done at compile time) plus the optimizations done by the native compiler.

All Memory access goes through the TARMProcessor::ManagedMemory* calls. If this call should fail (no memory, wrong permissions), the code throws an exception, abandoning simulation, and returning to emulation, then emulating the memory access fail vector. This rarely happens, so simulation still seems viable.

Also, if for whatever reason the simulator can't continue, the code falls back to emulation. For that reason, even simulated code need to keep the ARM stack as if it was emulated.

Now for the MMU. Simulation does not bother with the MMU yet. The usage of the MMU is so frigging complex that I really have a hard time simplifying it. The MMU can create pages as tiny as 1k bytes large, and every one of those pages can have access permissions that need to be checked on every memory access. These permissions are not only used to differentiate between system and user code - that would be easy, but also to generate "create-on-access" pages.

Example: you have some huge app (like 100 kilobytes or even *more* ;-). The app is store in a compressed format in flash. When the OS opens the app, *none* of that code is loaded. Instead, enough pages that *could* hold the uncompressed app are marked as "fail when accessed". Now whenever any of these addresses is read, the corresponding page is decompressed and put into RAM. When memory goes low, that page can be freed again, making more RAM available for variables - even though the app is still running.

As fascinating and awesome as this is, as long as we still have active code that requires this feature, we can not reduce MMU emulation complexity.

The Simulator is meant to have "C" code instead of an emulated ARM, so we can use regular debugging to find and recode these features.

8: FP emulation and SWIs. In the next commit, SWIs and FP functions can be called. They will currently cause a fallback into emulation, but that could easily be changed.

9: I have now finally figure out how exactly the task switching works in NewtonOS. It's simple and genius as the rest of the ROM. As soon as I can convert that into native code, simulated code can stay entirely in simulation, never entering emulation - assuming all functions of that thread are simulated. There are seven or so threads in the OS, plus the interrupts, which are basically a real-time thread.

10: It's wonderful to see developers getting active again! Have fun! Ask me if you think I can help. Also, EMail me directly if I appear to be ignoring the list: mm at matthiasm.com