It is the 1707th of March 2020 (aka the 1st of November 2024)
You are 3.94.202.151,
pleased to meet you!
mailto:blog-at-heyrick-dot-eu
Walking the RMA
There are many little parts of RISC OS that aren't so well understood. Documentation on these is scarce. I wanted to understand the Module Area (RMA) better, so I set about picking the RMA apart to get to know more about how it works.
A long time ago, a big clue was given to me by an errant module that was writing data beyond its allocation. This corrupted some things, leading subsequent actions on the module area to fail with a message such as "Not a heap block". I don't remember the exact text, but something came together in my head.
The truth is that the RMA is a large resizable memory zone (old Archimedes) or Dynamic Area (RiscPC and later) which contains a chained list of OS_Heap blocks. The blocks are laid out such that the word immediately prior to the block pointer gives an offset that points to the next object except for if this is a "free block" in which case the word pointed to gives the size of the free block (and is thus implicitly the pointer to the next block) which the regular offset pointer points to the next free block.
Yes, you have to keep track of both in tandem.
The end of the list is supposed to be denoted by the pointer being zero to mark the end of the list. I have seen it as zero, but I have also seen it as wildly bogus values. I guess I'm maybe reading something wrong here, I don't know. So I terminate the list if the pointer is zero, if the eventual address is beyond the end of the RMA, or if the eventual address is a negative number (as would be the case of adding &80000000 or more to the pointer - integers are signed in BASIC).
You might be saying "Hey, wait, Reporter can do this!". You're right, Reporter can, in a nice clicky-pointy sort of way too. However we have two benefits:
Because of Reporter's user interface, there's no way you can easily see four thousand blocks listed. It can be done, by turning on the logging and restarting Reporter, but it takes about four times as long as typing RMAwalker { > $.rmastuff } and letting my program dump the items which RISC OS then redirects output to a file on disc.
Reporter doesn't attempt to be intelligent. It just lists allocated and freed blocks. My RMAwalker first collects the addresses of modules and their workspace and when listing objects it will try to match an address to indicate what is there.
The last point, however, is somewhat inadequate. If you have a module written in C, the C environment will claim some workspace and RMAwalker will duly list this. However, if you then go and malloc() a few times, these blocks will also be claimed from the RMA, however there is no sensible way to associate these additional blocks with a particular module other than by association.
Consider:
It is quite likely that the three claims following BufferManager are indeed related to it, but this becomes a lot harder to work out following system initialisation where all sorts of junk is present in the RMA.
Don't take my word for it - here's the top and tail of my RMA dump:
As the end summary indicates, there are three and a half thousand entries in my RMA. Note that the sizes are slightly different - there's a four byte disparity with the size of the largest free block and 320 bytes disparity in the free size count. This is because I'm counting OS_Heap blocks. The module area does things a little differently (where do you think the module's "Private Word" is stored, for instance?).
Now on to some code. Yay!
I won't be annotating the code, the comments should be sufficient. Copy-paste the program code into !Edit, set the file's type to be "BASIC" and then save the program as "RMAwalker".
REM >RMAwalker
REM Walk the RMA
REM
REM By Rick Murray, 2016/08/01
REM
REM Open Source software - this code has been released under the EUPL v1.1 (only).
REM https://joinup.ec.europa.eu/community/eupl/og_page/european-union-public-licence-eupl-v11
REM
REM Allocate memory for up to 256 modules
DIM modbase%(256)
DIM modwksp%(256)
DIM modname$(256)
modmax% = 0 : REM How many modules we found
alloccount% = 0 : REM Count of allocated blocks
freecount% = 0 : REM Count of free blocks
allocsize% = 0 : REM Size of allocated blocks
freesize% = 0 : REM Size of free blocks
DIM buffer% 15 : REM Buffer for pretty number formatting
REM Zero module data blocks
FOR recurse% = 0 TO 255
modbase%(recurse%) = 0
modwksp%(recurse%) = 0
modname$(recurse%) = ""
NEXT
REM Get RMA boundaries (RMA is DA 1; this will fail on old RISC OS)
SYS "OS_DynamicArea", 2, 1 TO ,, rmasize%, rmabase%
rmaend% = rmabase% + rmasize%
SYS "OS_Module", 5 TO ,,rmalargest%, rmafree%
REM Header text
PRINT "Examining Module Area"
PRINT
PRINT "RMA base : &"+STR$~(rmabase%)
PRINT "RMA size : "+STR$(rmasize%)+" bytes"
PRINT "RMA end : &"+STR$~(rmaend%)
PRINT "Free space: "+STR$(rmafree%)+" bytes";
PRINT " (largest block is "+STR$(rmalargest%)+" bytes)"
PRINT
REM Read module information
REM We only look at first 255 modules so we don't risk overrun
FOR recurse% = 0 TO 255
SYS "XOS_Module", 12, recurse% TO ,,,modptr%, modpw% ; f%
IF ((f% AND 1) = 0) THEN
REM Didn't fault, so we have module information
valid% = FALSE
REM Is the module itself in the RMA? (otherwise ROM)
IF ( (modptr% > rmabase%) AND (modptr% < rmaend%) ) THEN
modbase%(modmax%) = modptr%
valid% = TRUE
ENDIF
REM Is the workspace in the RMA? (otherwise a DA, no workspace, etc)
IF ( (modpw% > rmabase%) AND (modpw% < rmaend%) ) THEN
modwksp%(modmax%) = modpw%
valid% = TRUE
ENDIF
REM If valid is TRUE, then module and/or workspace is in RMA, so retrieve modname
REM (icky hack time)
IF (valid% = TRUE) THEN
SYS "XOS_GenerateError", (modptr% + modptr%!16) TO name$
modname$(modmax%) = name$
modmax% += 1
ENDIF
ENDIF
NEXT
REM Verify the heap is a heap (paranoid!)
IF (rmabase%!0 <> &70616548) THEN ERROR 1, "RMA not a Heap?"
REM Calculate "first free block"
rmanextfree% = (rmabase% + rmabase%!4 + 8)
REM Try walking the heap...
this% = rmabase% + &14
REPEAT
nextptr% = this%!-4
isfree% = FALSE
endoflist% = FALSE
REM Fudge things if this is a free block
IF (this% = rmanextfree%) THEN
REM PRINT this%!-4, this%!0, this%!4
rmanextfree% += nextptr% : REM Set free pointer to next free object
nextptr% = this%!0 : REM Set next item appropriately here
isfree% = TRUE
ENDIF
REM If next pointer is zero, we'll need some more fudging.
IF (nextptr% = 0) THEN
endoflist% = TRUE
nextptr% = (rmaend% - this%) + 4
ENDIF
REM More fudging if nextptr% is bogus and following it will make us blow up
REM Sometimes we don't always reach the end of the heap tidily.
REM Not sure what the difference is, suffice to say that Reporter may see an
REM unused block of 40,848 bytes in size. We, however, are likely to see the
REM same as two blocks - in this case 4,128 bytes and 36,720 bytes; with the
REM nextptr being something gibberish.
test% = this% + nextptr%
IF ( (test% > rmaend%) OR (test% < 0) ) THEN
REM Same as just above
endoflist% = TRUE
nextptr% = (rmaend% - this%) + 4
ENDIF
REM It's an allocation unless we know otherwise
alloccount% += 1
allocsize% += nextptr%
REM Print what we've found (up to "9,999,999")
PRINT "&"+STR$~(this%)+" : ";
PRINT RIGHT$(" "+FNprettynum(nextptr%), 9)+" ";
REM Is this a module?
FOR recurse% = 0 TO (modmax% - 1)
IF (modbase%(recurse%) = this%) THEN
PRINT "Module : "+modname$(recurse%);
ENDIF
NEXT
REM Is this workspace?
FOR recurse% = 0 TO (modmax% - 1)
IF (modwksp%(recurse%) = this%) THEN
PRINT "Workspace : "+modname$(recurse%);
ENDIF
NEXT
REM Is it a free block?
IF (isfree% = TRUE) THEN
PRINT "Free block";
alloccount% -= 1
allocsize% -= nextptr%
freecount% += 1
freesize% += nextptr% : REM Fudge factor
ENDIF
REM Is this the end of the list?
IF (endoflist% = TRUE) THEN
PRINT "Unused space";
alloccount% -= 1
allocsize% -= nextptr%
freecount% += 1
freesize% += nextptr%
nextptr% = 0 : REM Reset next object pointer to zero
ENDIF
REM Give a linefeed to end whatever printed
PRINT
REM Point to the next item
this% += nextptr%
UNTIL (nextptr% = 0)
REM Print a summary.
REM Note that our free block bytes will NOT match the value returned by OS_Module.
PRINT
PRINT "Listed "+STR$(alloccount%)+" allocated blocks totalling "+FNprettynum(allocsize%)+" bytes,"
PRINT "and "+STR$(freecount%)+" free blocks totalling "+FNprettynum(freesize%)+" bytes."
END
DEFFNprettynum(what%)
SYS "OS_ConvertSpacedCardinal4", what%, buffer%, 16 TO ,term%
term%?0 = 13
FOR recurse% = buffer% TO term%
IF recurse%?0 = 32 THEN recurse%?0 = 44
NEXT
=$buffer%
Some ideas (exercises for the reader):
Keep scanning modules until we run out of allocation space - not all modules are in the RMA or have workspace in the RMA (SharedCLibrary, for example).
The address matching is horribly inefficient. It scans each list of addresses entirely; if a match is found it should stop at that point. Probably easiest to do by making that bit of code a PROC so you can just ENDPROC when you're done.
And finally? You'll be surprised at exactly how much rubbish is in the RMA. And a little depressed that Acorn's OS_Heap SWI never included a mechanism to walk ("enumerate") the heap. Plus it is a shame that the RMA code doesn't tag claims with a backpointer (return R14?) to what made the claim; but back in '85 these sorts of things were likely not thought of and besides memory was expensive. Three and a half thousand blocks, a word extra apiece, that's about 14KiB. That could be a lot on a 1MiB machine.
Your comments:
Please note that while I check this page every so often, I am not able to control what users write; therefore I disclaim all liability for unpleasant and/or infringing and/or defamatory material. Undesired content will be removed as soon as it is noticed. By leaving a comment, you agree not to post material that is illegal or in bad taste, and you should be aware that the time and your IP address are both recorded, should it be necessary to find out who you are. Oh, and don't bother trying to inline HTML. I'm not that stupid! ☺ ADDING COMMENTS DOES NOT WORK IF READING TRANSLATED VERSIONS.
You can now follow comment additions with the comment RSS feed. This is distinct from the b.log RSS feed, so you can subscribe to one or both as you wish.
Anon, 3rd August 2016, 10:35
The RMA... what a horrible hack. I guess in the days of 1MB RAM and the system being used for a bit then switched off it wasn't a problem. The computer got rebooted before the RMA got fragmented.
By the time I had a RISC PC with 128MB of RAM, RMA fragmentation had become a serious problem. Most newer apps used dynamic areas where they were available. A few older ones didn't. Those apps cause the problem.
(I'm writing this on a PC with a 6-core 64-bit CPU, 32GB of RAM and a 1TB SSD... how times change!)
David Pilling, 10th August 2016, 02:49
Any chance of taking an off the shelf (open source etc) memory manager and producing a new module. I set off reading this thinking I'd been involved at some point, but I think that was with how C malloc works. In my efforts I never used OS_Heap - unless there was no alternative or you're about to point out that C malloc just passes everything on to OS_heap.
This web page is licenced for your personal, private, non-commercial use only. No automated processing by advertising systems is permitted.
RIPA notice: No consent is given for interception of page transmission.