Thanks for taking the time to do that, I'm off to have a read now, as it's an area you're aware I'm very involved in.

If you don't mind, I'll post any questions publicly on here, should anyone else also be interested in the technical side of what you've achieved.

May I ask, what are your goals?

the reader should understand that the development of Aemulor has been split almost 50/50 between the RO4 API emulation and the 26-bit CPU emulation. Contrary perhaps to the expectations of most people, emulating a 26-bit CPU on the 32-bit XScale has not been the most difficult part of developing Aemulor.

SWIs and CPU flags / SWI API changes

The 32-bit version supplied with the IYONIX pc does not provide support for the 26-bit procedure calling standard (APCS-R) used by 26-bit C programs. Aemulor therefore loads the 26-bit version that is supplied with the Castle C/C++ Development Suite, and hides the 32-bit one so that 26-bit applications see only the 26-bit SharedCLibrary.

Aemulor RMA
...
From RISC OS's perspective, the Aemulor RMA is a normal dynamic area, but Aemulor remaps the memory at an address below 64MB so that it becomes addressable within the 26-bit environment. Because this emulated RMA is visible to all applications, native 32-bit applications are also restricted to a maximum size of 28MB each (as per RISC OS 4) whilst Aemulor is running. It is hoped that this limitation can be removed with a later version.

A Simple Interpreter

Also, in practice, most self-modifying code actually modifies code that follows the current position (PC) rather than preceding it.

A Faster Interpreter

Hardware Breakpoints

Just In Time Compilation

Sound & Filing Systems
...
Emulating 26-bit voice generators poses some interesting technical challenges because Aemulor itself must be capable of running in IRQ mode with IRQs enabled

Screen mode emulation

The emulated screen modes are implemented by telling RISC OS to use a different area of memory for the screen image, so that the full graphics capabilities of the RISC OS kernel are still available, including scrolling and bankswitching. This ensures much better compatibility than would be achieved by intercepting all graphics operations and reimplementing them ourselves.
..
This conversion is done up to 15 times per second to ensure a smooth update, which is particularly important for games. A future version of Aemulor Pro may use a different approach to screen updating but the idea is currently unproven so I won't describe it here.

Hardware emulation

One hardware feature currently not emulated by Aemulor Pro is the screen configuration registers of the VIDC and VIDC20 devices. These are manipulated by a small number of games to achieve some scrolling effects, so it's anticipated that they too will be emulated in a future release of Aemulor Pro.

Just In Time Compilation
..
The inclusion of compilation in Aemulor Pro is particularly exciting because it opens up the possibility of many more optimisations such as substituting XScale-tuned equivalent code or increasing the performance of floating point maths by avoiding the repeated decoding of instructions. Whilst it should be stated that JIT compilation will not work for all applications, it is expected to work for the vast majority of StrongARM-aware programs, and the other emulation engines will always be available for those applications which cannot be used with the compilation.

Task display

Improved handling of low-resolution screen modes
..
To solve these problems, Aemulor Pro will in future be capable of rescaling the screen image in software so that the monitor can be operated in a suitable higher-resolution, higher frame-rate mode, whilst the game sees the resolution and frame rate that it expects. The user interface will be extended to allow the user to specify which native screen mode should be used for each emulated mode.

Control of the emulation speed

Windowed emulation of single-tasking applications

SWIs and CPU flags / SWI API changesThis is an area I've really struggled with, I've been unable to find the kind of detail I require to implement this to date. I've reached out to the community for assistance, but have sadly had no responses. If you have any documetation on the subject you could share, I'd really appreciate it.

The circular buffer is a very neat idea, how did you determine 32 instructions as being an optimal size?

Legacy C code is a fine example here, where SWI instructions are written to memory prior to execution as CallASWI didn't exist back then.

Do you cache the breakpoint locations so it's not continually scanning ahead?

Avoiding the cache flush is a stroke of genius. ADFFS has to flush as I've opted to make no processor specific optmizations past SA, so it's future proofed. I've yet to get the Iyonix to flush correctly though, even calling OS_SynchroniseCodeArea results in it writing utter rubbish to memory.

Do you cache any JIT sequences? Or are they disposed of once executed?

I'm interested to understand why you went with 8bpp.

I'm interested to know how much you ended up emulating, did you go for full emulation or just what was required. Do you for example provide T0, T2 and T3

I don't expect any software written for the RiscPC to be touching hardware directly, games written for RiscPC seem to only touch VIDC20 and T1 directly which is already covered by ADFFS.

How far did you get with this and how were you planning to resolve the self-modifying code problem?

What was the deciding factor of using a seperate Task display vs letting RISCOS display them in the regular Task manager?

Control of the emulation speedI'm interested to know what methods you used here. As mentioned above, ADFFS in the most part stalls at OS_Byte 19 / 112 and OS_Word 21, 0 although I'm looking at other techniques for games that rely on slow processor speed alone.

This is something I plan to implement via a RISCOS Hypervisor.

I rather suspect that your requirements will be somewhat different to those of Aemulor, because one key design decision in the current versions of Aemulor was never to change an address; all code and data is deliberately kept at the address for which it was built. This both speeds up the emulation, and sidesteps the massive undertaking of having to relocate all addresses passed to 32-bit code, including those contained within data structures.

If it's any help, SWIs that Aemulor treats specially may be grouped into the following categories:

- SWIs which need special handling of CPU flags upon return, because they actually need to return information via the flags (I ascertained this list by reading through the PRMs from start to finish!)

- SWIs which have had their API redefined because they did not accept a full 32-bit address in one or more registers; rather they used some bits of the register for another purpose (OS_ReadLine, OS_HeapSort, OS_SubstituteArgs, OS_File)

- SWIs which accept the address of some code, and thus must be intercepted by Aemulor (so that it can step intercept callsbacks into 26-bit code; may not affect ADFFS). I think there are far more SWIs in this category.

Legacy C code is a fine example here, where SWI instructions are written to memory prior to execution as CallASWI didn't exist back then.

True, but I presume that you've realised the SWI value is picked up from the instruction by a data read, so there's no need to clean caches in this case, even on the StrongARM/XScale Harvard architecture. This is also true for FP instructions, provided that all the FP maths is being performed by the FPEmulator (ie. no FPA), but you must ensure that the slot is pre-filled with a instruction that is guaranteed to raised an Undefined Instruction exception.

Do you cache the breakpoint locations so it's not continually scanning ahead?

Aemulor has 3 engines; the ARM610-compatiable interpreter engine and the StrongARM-compatible breakpointing approach both make very heavy use of cacheing to improve performance and avoid instruction scanning/decoding. In fact I recently measured the instruction execution:decode ratio as 3400:1. The third engine, labelled the 'ARM3' engine uses minimal cacheing to achieve maximum compatibility with older, less well-behaved software.

I've yet to get the Iyonix to flush correctly though, even calling OS_SynchroniseCodeArea results in it writing utter rubbish to memory.

Avoiding the expensive cache clean (technically different from a flush, in ARM terminology) was another key design decision, for performance reasons.

I'm unaware of any problems with the OS_SynchroniseCodeAreas SWI, and I think a lot of stuff would be broken if there were an implementation fault, so - with respect - it's probably something in your code. One oddity of this API call is that the end address is inclusive.

Do you cache any JIT sequences? Or are they disposed of once executed?

I don't really have a current JIT engine. To be honest the prototype that I wrote early in Aemulor's development was found to have disappointing performance, and it was then superceded by the breakpointing approach. It's something that I may revisit for later CPU architectures that do not make available the debug facilities that Aemulor requires.

Re IRQ handlers:

What were the challenges?

The chief complications here are handling reentrancy; if you have even a single instruction out of place and don't consider that your engine can be interrupted and re-entered at any point, then you will get crashes, and they will not be fun to track down. There is the additional complication that IRQ mode register(s) are corrupted by the execution of interrupt handlers on RISC OS; SPSR_irq certainly, and you may want to check R14_irq as well (I forget about that one). Your reply, however, suggests that you may already be aware of the details.

I'm interested to understand why you went with 8bpp.

Quite simply performance; pushing less data out to the graphics cards, and the 256-entry palette can obviously represent all the colours required for lower-depth modes. The main requirement for low-bpp modes in Aemulor was supporting Sibelius, which maintains the current desktop mode, rather than using a lower-resolution custom modes like games. This means that the amount of data being shuffled around can be enormous. (This monitor is running at 2048 x 1280, for example!)

I'm interested to know how much you ended up emulating, did you go for full emulation or just what was required. Do you for example provide T0, T2 and T3

I think I'm right in saying that the IYONIX pc offers only two timers via its HAL_ interface. A quick perusal of the source shows that I only support timers 0 and 1, but that has never been a problem.

I don't expect any software written for the RiscPC to be touching hardware directly, games written for RiscPC seem to only touch VIDC20 and T1 directly which is already covered by ADFFS.

I do implement emulation of most IOMD/IOC registers; they are very similar devices, but remember that my main focus was RiscPC-era software, so I considered IOMD more important, and IOC compatibility a 'bonus.'

Useful resources, by the way, which you may be able to pick up on ebay, for example. I'm sure that they're all out of print now:

Acorn RISC Machine Family Data Manual (VLSI Technology Inc, Prentice Hall, ISBN 0-13-781618-9)

Technical Reference Manuals for A3000 and RiscPC.

Self-modifying code isn't all that common in the applications that people want to run under Aemulor, because they have for the main part been updated for StrongARM-compatibility at least. It would be supported, but I do not expect it to be inexpensive in terms of performance.

The route I took was to change memory pages to Read only if they contained an instruction and handle writes/self-modifying code in an Abort handler. It's faily impressive to watch the Pi handling millions of Aborts a second - although obviously not the most optimal approach, it doesn't seem to impact game speed. Zarch for example will run at 700fps unthrottled and trigger 1.1m Abort/sec.

Am I to presume that you then have a problem with self-modifying priviledged-mode code? If you're performing the write by proxy in an abort handler, that is presumably only for USR mode aborts. If you are actually changing the page attributes and then performing the write, you'll find a limited-range cache clean of the instruction(s) is surely cheaper, because otherwise the OS will have to clean the page tables out to memory, as well as flushing the TLBs.

Control of the emulation speed - I'm interested to know what methods you used here. As mentioned above, ADFFS in the most part stalls at OS_Byte 19 / 112 and OS_Word 21, 0 although I'm looking at other techniques for games that rely on slow processor speed alone.

To be honest, I don't think I concentrated upon that too much effort, adopting the attitude that well-written games would by VSync-driven using those OS_Byte calls, and that provided suitable (/close-enough) screen modes are employed, with the appropriate frame rate, most games should be okay.

This is something I plan to implement via a RISCOS Hypervisor.

I wish you luck and hope that you have the time. I, alas, do not have the luxury of lavishing extended amounts of time on RISC OS software development, although I too have an interest in hypervisors, having just been reading up on the features of the later ARM architecture versions, including their VM support.

It's undoubtedly my code, but I can't fathom out why it only affect the Iyonix - it's the same codebase across StrongARM, Iyonix and Pi so they should all act the same.

As you point out, IOC/IOMD overlap so I probably don't need to add much specific IOMD support to get later software running.

Back in 1988/89 I received a lot of assistance from Acorn/VLSI whilst coding Podule drivers and sample playback routines, so have an original 1st print of the VLSI manual.

If a Pi comes out with ARMv8.1 - then I'll look at upping the minimum requirement.

The XScale CPU has 32KB each for I and D cache, and the other two have only 16KB each. The cache lines may also be longer, but I don't recall off-hand. Is there anything in your code that assumes the cache properties such as rounding of addresses?

As you point out, IOC/IOMD overlap so I probably don't need to add much specific IOMD support to get later software running.

That's understating it somewhat. IOMD is custom logic that implemented IOC and the DRAM-related MEMC functionality (which the ARM610/other does not provide) but not the logical->physical address mapping of MEMC. Thus the required software effort was reduced

If a Pi comes out with ARMv8.1 - then I'll look at upping the minimum requirement.

I wouldn't hang around for that; that would seem to be serious overkill for the current applications of the Pi, and I think the required development work would be prohibitive.