[John, Landon, Walter and Greg: I cc'ed you because it would be nice
if you could explain me XIPObjects and Transactions: see at the end
Summary of the previous episodes for our Ex Newton Friends:
I've implemented an ATA driver. The driver itself works well. I have
implemented partition read for DOS and a new format to have several
stores on a single card since the Flash card limit is 32 MB while
there are 64 MB linear cards. Hence I guessed that having several
stores may be preferrable to unique big store. (there are also Newton
RAM limits). I also implemented the card handler and the NewtonScript
interface so that when the card is inserted a slip is open with the
partition map.
I am trying to interface the driver with the NewtonOS. I have
therefore looked at what already exists, TPackageStore and
TFlashStore (and TMuxStore, but this seems to be there just for the
monitor)]
Hi all,
after having worked on the Newton Storage system by reverse
engineering large parts of TFlashStore and TPackageStore (and
associate classes) I have come to the following conclusion:
I need to implement my own store format. I can't technically use
TFlashStore or TPackageStore to access the data. I can't use their
format because:
a/ TFlashStore's format is too complex and I haven't reversed
engineered it entirely. It is made for 64 KB blocks because the chips
use 64 KB erasable blocks, which is not the case of ATA flash. (each
sector can be written independantly)
b/ TPackageStore is made for read-only stores (it is basically a
table of content of each object and immediatly afterward the objects
themselves added one after the other without any alignment)
Stores are structures that store objects. Objects are referred by an
ID (which the store allocates itself) and a size (which is
modifiable). Objects on FlashStore cannot be larger than 64 KB minus
one byte (this is funny because 65534 bytes long objects won't fit on
a 64 KB block because of other informations, but anyway, I was able
to allocate such objects).
Hence I guess I only have to create an interface to stores those
objects, delete them, read and write them, change their size,
allocate new objects. I already know how to mount and unmount a store
to have it used by NewtonScript. I am working on how to find the
object ID which contains the name of the store to display it on the
interface before it is mounted (once mounted it is just aStore.name).
I think it is accessed the following way:
RootObject starts with 'WALY' (57414C59) then 4 blocks ID.
The last one points to a block with:
FFFFFFFF FFFFFFFF
name block ID
various length data
The block with the name block ID is a short for the string size and
then a CString.
In fact, the system handles the store itself, that's its problem, not
mine. But I cannot change things on the store in its back.
I haven't found anything which explains the 32 MB limit. I have found
IDs rather large (03xxxxxx). 02000000 is 32 MB, hence if each object
is two bytes long, with IDs from 0 to 03000000, we'll have more than
32 MB. In fact, the 32 MB limit seems to be internal to TFlashStore
and I may design a store format with a larger limit.
Here is the ontological question:
considering that objects can have a random size between 6 (what I
have observed, but 1 may also occur) and 64 KB,
considering that there may be some crash problems (don't know how it
is handled on the Flash, but it may be a good idea to have no data
waiting too long to be written)
considering that the minimum (and unique) unit of transation of data
with the ATA disk is a sector (512 bytes)
considering that it may be a good idea to maintain a table of damaged
sectors (provided that CF-ATA offers a function to get the number of
writes on a sector, and that I may offer test utilities to map out
damaged sectors)
which format to use?
i.e. what is the minimum allocated size for objects?
objects should include the size. If the ID is translated to the
position on the disk without a map on the disk (faster) I can't
defragment the disk. If the ID is in table(s), I could, but this is
slower. And the table may need to be written everytime I create a new
object.
If I allocate a minimum size smaller than 512 bytes, everytime a
small object is changed, I'll write the whole sector (remember that
flash technology have a maximum number of writes). Objects that have
been resized may be on several sectors (fragmentation) which may lead
to slow access.
If I allocate a minimum size of 512 bytes, there will be great loss
of space for small objects.
The more I think about it, the more the table ID -> position on disk
seems to be the best solution, but this will take additional sectors.
The performance may be really bad with it.
Should I really don't maintain a write cache? (if you remove the
power, there is no hope to recover the data which may be corrupted at
any level).
Once this ontological question is solved (read: once you told me what
you think about it), I'll design the format of TATAStore. Sure, this
will be published, but what you'll need to extract data is to
understand what the system does with all those objects.
I haven't found everything in the interface to implement a complete
store object. In fact, I haven't understood exactly what XIPObjects
were and how transations were used. Those are for Flash only,
PackageStores ignore them or returns the appropriate error to say it
is not supported.
More details about those methods I don't know about:
== Transactions ==
NewtonErr NewWithinTransaction( long inSize, unsigned long*
outObjectID );
// TPackageStore calls NewObject( long, unsigned long*)
NewtonErr StartTransactionAgainst(unsigned long);
NewtonErr SeparatelyAbort(unsigned long);
NewtonErr AddToCurrentTransaction(unsigned long);
NewtonErr InSeparateTransaction(unsigned long);
// TPackageStore returns noErr for those four functions.
Boolean InTransaction( void ); // Not in the JT's symbols !
// TFlashStore returns mIsIntransaction.
== NewObject(char*, long inSize, unsigned long* outObjectID): the
first argument ==
NewtonErr NewObject(char*, long inSize, unsigned long* outObjectID);
// What is the first argument? TFlashStore::NewObject( long,
unsigned long*) calls this with
// nil for the first argument. A name?
== XIPObjects ==
NewtonErr CalcXIPObjectSize(long, long, long*);
// TPackageStore returns kError_XIP_Not_Possible
NewtonErr NewXIPObject(long, unsigned long*);
// TPackageStore returns kError_XIP_Not_Possible
NewtonErr GetXIPObjectInfo(unsigned long, unsigned long*,
unsigned long*, unsigned long*);
// TPackageStore returns kError_XIP_Not_Possible
Regards,
Paul
-- P&M Consulting Newton Program http://www.pnm-consulting.com/newton/ *************************************** NewtonTalk brought to you by:EVOTE.COM -- the ESPN of politics on the Internet! All the players, all the news, and the hottest analysis and features (plus 'toons!) anywhere.... visit http://www.evote.com today!
*************************************** Need Subscribe/Unsubscribe info?
Visit the NewtonTalk section at http://www.planetnewton.com
This archive was generated by hypermail 2b29 : Tue Aug 01 2000 - 06:50:08 CDT