For this first article looking at how to create PackMan/RiscPkg packages, I've decided to use my SunEd program as a guinea pig. Being a simple C application with no dependencies on other packages, it'll be one of the most straightforward things on my site to get working, and one of the easiest for other people to understand.
 
Read on to discover how to turn simple apps like SunEd into RiscPkg packages, and more importantly, how to automate the process.
 

Building your first package, the PackIt way

The RiscPkg policy manual is a rather dry document, so the easiest way of getting your first package built is to use the PackIt tool created by Alan Buckley. After loading PackIt, you can simply drag an application to its iconbar icon and up will pop a window to allow you to enter all the extra details that RiscPkg needs to know.

Once everything is filled in correctly, opening the menu and selecting the "Save" option should allow you to save out the resulting package zip file.

... except that the current version of PackIt seems to save it out with the wrong filetype. No problem, just manually set the type to 'zip' or 'ddc' and things look a lot better:

Pretty simple, isn't it? The !SunEd app has been placed inside an Apps.File directory (mirroring the default install location for the application on the user's hard disc), while the information that was entered into PackIt's wizard has been saved to the RiscPkg.Control and RiscPkg.Copyright files.

Control is a simple text file containing the package metadata (the structure of which is the subject of much of the RiscPkg policy document), while Copyright is a verbatim copy of the copyright message you entered into PackIt's window.

Now that you have a package built, you can easily test it out by dragging it to PackMan's iconbar icon. PackMan will then go through the usual installation procedure, just as if it was a package you'd selected to install from the Internet.

Automating package building

Filling in PackIt's wizard once the first time you create a package for an app is all well and good, but what about when you want to release an update for the package? Entering the information all over again is going to waste your time and introduce the risk of making mistakes.
 
Most C/C++ developers are already familiar with using makefiles to build their programs. With a bit of effort, it's possible to create makefiles which can also automate creation of the corresponding RiscPkg package.

Before

After a brief bit of preparation, the 2003-vintage SunEd sources were tidied up and a simple makefile was written, allowing the application binary to be easily rebuilt on command.

CFLAGS = -Wall -mpoke-function-name -O2 -mlibscl -mthrowback -static   CC = gcc -c $(CFLAGS) -MMD LINK = gcc $(CFLAGS)   SRCS = \  suned \  limp   OBJS = $(addsuffix .o, $(SRCS))   # Output file !SunEd/!RunImage: $(OBJS)     $(LINK) -o $@ $^ -mlibscl   # Object files %.o: %.c     $(CC) -MF d/$(basename $@) -o $@ $<   # Dependencies -include d/*

The original SunEd makefile

 
As a brief overview:

• c and h contain the source code as you would expect
• d and o are used for intermediate files: autogenerate dependencies and object files
• !SunEd is the full app, ready for distribution, and the makefile is only used to rebuild the !RunImage

And after

Rather than bore you with all the intermediate versions, I figured it was best to just jump straight to the final version of the makefile and the adjusted source structure.

CFLAGS = -Wall -mpoke-function-name -O2 -mlibscl -mthrowback -static   CC = gcc -c $(CFLAGS) -MMD LINK = gcc $(CFLAGS)   CP = copy CPOPT = A~CF~NQR~S~T~V   SRCS = \  suned \  limp   APP = Apps/File/!SunEd   ROAPP = $(subst /,.,$(APP)) OBJS = $(addprefix build/,$(addsuffix .o, $(SRCS)))   # Output file build/!RunImage: $(OBJS)     $(LINK) -o $@ $^ -mlibscl   # Object files build/%.o: src/%.c build/dirs     $(CC) -MF build/d/$(subst /,.,$(basename $@)) -o $@ $<   # Pattern rule for injecting version numbers into files build/%.sed: src/template/% src/Version build/dirs     sed -f src/Version $< > $@   # Explicit dependency needed for generated file build/VersionNum.sed build/suned.o: build/VersionNum.sed   # Standard clean rule clean:     remove binary/zip     remove source/zip     x wipe build ~CFR~V   # Binary RiscPkg archive binary.zip: build/pkg-dir     remove binary/zip     dir build.pkg     zip -rqI9 ^.^.binary/zip *     dir ^.^   # Source zip archive source.zip: build/src-mani makefile COPYING     remove source/zip     zip -rqI9 source/zip src makefile COPYING   all: binary.zip source.zip   build/dirs:     cdir build     cdir build.o     cdir build.d     create build.dirs   # Double-colon rules execute in the order they're listed. So placing this rule # here makes sure that the 'build' folder exists prior to the rule below being # executed. build/pkg-mani:: build/dirs   # Double-colon rules with no pre-requisites always execute. This allows us to # make sure that build/pkg-mani is always up-to-date build/pkg-mani::     src/manigen src.pkg build.pkg-mani   # Same system as build/pkg-mani build/src-mani:: build/dirs build/src-mani::     src/manigen src build.src-mani   # Create the package dir ready for zipping build/pkg-dir: build/pkg-mani build/!RunImage build/Control.sed build/!Help.sed COPYING # Copy over the static files     x wipe build.pkg ~CFR~V     $(CP) src.pkg build.pkg $(CPOPT) # Populate the RiscPkg folder     cdir build.pkg.RiscPkg     $(CP) build.Control/sed build.pkg.RiscPkg.Control $(CPOPT)     $(CP) COPYING build.pkg.RiscPkg.Copyright $(CPOPT) # Populate the app folder     $(CP) build.!Help/sed build.pkg.$(ROAPP).!Help $(CPOPT)     $(CP) build.!RunImage build.pkg.$(ROAPP).!RunImage $(CPOPT) # Create the dummy file we use to mark the rule as completed     create build.pkg-dir   # Dependencies -include build/d/*

The new SunEd makefile

 
As you can see, there have been a fair number of changes. Not all of them are strictly necessary for automating package creation (after all, a package is little more than a zip file), but this structure has resulted in a setup that helps to minimise the amount of work I'll need to do when preparing new releases. The setup should also be easily transferrable to the other software I'll be wanting to package.

What it does

• The clean rule reduces things to the state you see above
• The source.zip rule builds a source archive, containing exactly what you see above
• The binary.zip rule builds the RiscPkg archive, performing the following operations to get there:
  
  • A copy of the src.pkg folder is made, in order to provides the initial content of the package zip - essentially, the static files which aren't modified/generated by the build.
  • As you'd expect, the !RunImage file gets built and inserted into the app. But that's not all!
  • The src.Version file is actually a sed script containing the package version number and date:
    
     
    

    s/__UPSTREAM_VERSION__/2.33/g s/__PACKAGE_VERSION__/1/g s/__RISCOSDATE__/28-Aug-18/g

    The src.Version file

    
     
    This sed script is applied to src.template.!Help to generate the help file that's included in the package, src.template.Control to generate the RiscPkg.Control file, and src.template.VersionNum. By driving all the version number / date references off of this one file, there won't be any embarrassing situations where a built program will display one version number in one location but another version number in another location.
  • src.template.VersionNum is a C header file, which is used to inject the app version and date into !RunImage.
  • The COPYING file in the root used as the RiscPkg.Copyright file in the package.
• All the intermediate files will be stored in a build folder, which helps keen the clean and source.zip rules simple.
• Full dependency tracking is used for both the source.zip and binary.zip targets - adding, removing, or changing any of the files in src.pkg (or anywhere else, for source.zip) will correctly result in the resulting target being rebuilt. This is achieved without introducing any situations where the targets are redundantly built - so a build system which tries to build tens or hundreds of packages won't be slowed down.

manigen

There are also a few extra files. The src.notes folder is a collection of notes from my reverse-engineering of the SunBurst save game format, which I've decided to include in the source archive just in case someone finds it useful. But that's not really relevant to this article.
 
manigen, on the other hand, is relevant. It's a fairly short and straightforward BASIC program, but it plugs a very large hole in make's capabilities: Make can only detect when files change, not directories. If you have a directory, and you want a rule to be executed whenever the contents of that directory changes, you're out of luck. For small projects like SunEd this isn't so bad, but for bigger projects it can be annoying, especially when all you really want to do with the files is archive them in a zip file.
 
Thus, manigen ("manifest generator") was born. All it does is recursively enumerate the contents of a directory, writing the filenames and metadata (length, load/exec addr, attributes) of all files to a single text file. However, it also compares the new output against the old output, only writing to the file if a change has been detected.
 

out%=0 ON ERROR PROCerror   REM Parse command line args SYS "OS_GetEnv" TO args$ REM First 3 options will (hopefully) be 'BASIC --quit ""' opt$ = FNgetopt : opt$ = FNgetopt : opt$=FNgetopt REM Now the actual args dir$ = FNgetopt out$ = FNgetopt   DIM result% 1024   out%=OPENUP(out$) IF out%=0 THEN out%=OPENOUT(out$) mod%=FALSE   PROCprocess(dir$) IF EOF#out%=FALSE THEN mod%=TRUE IF mod% THEN EXT#out%=PTR#out% CLOSE#out% REM Oddity: Truncating a file doesn't modify timestamp IF mod% THEN SYS "OS_File",9,out$ END   DEF PROCprocess(dir$)   LOCAL item%   item%=0   WHILE item%<>-1     SYS "OS_GBPB",10,dir$,result%,1,item%,1024,0 TO ,,,read%,item%     IF read%>0 THEN       n%=20       name$=dir$+"."       WHILE result%?n%<>0         name$=name$+CHR$(result%?n%)         n%+=1       ENDWHILE       PROCwrite(name$+" "+STR$~(result%!0)+" "+STR$~(result%!4)+" "+STR$~(result%!8)+" "+STR$~(result%!12))       IF result%!16=2 THEN PROCprocess(name$)     ENDIF   ENDWHILE ENDPROC   DEF FNgetopt   LOCAL opt$   opt$=""   WHILE ASC(args$)>32     opt$ = opt$+LEFT$(args$,1)     args$ = MID$(args$,2)   ENDWHILE   WHILE ASC(args$)=32     args$ = MID$(args$,2)   ENDWHILE =opt$   DEF PROCerror   PRINT REPORT$;" at ";ERL   IF out%<>0 THEN CLOSE#out% END   DEF PROCwrite(a$)   LOCAL b$,off%   IF EOF#out% THEN mod%=TRUE   IF mod%=FALSE THEN     off%=PTR#out%     b$=GET$#out%     IF a$<>b$ THEN mod%=TRUE : PTR#out%=off%   ENDIF   IF mod% THEN BPUT#out%,a$ ENDPROC

manigen

 
On Unix-like OS's this is the kind of thing you could knock together quite easily using standard commands like find, ls, and diff. But the built-in *Commands on RISC OS aren't really up to that level of complexity (or at least not without the result looking like a jumbled mess), so it's a lot more sensible to go with a short BASIC program instead.
 
The usage of manigen in the makefile is described in more detail below.

Makefile magic

Looking at each section of the makefile in detail:

Pattern rules

# Object files build/%.o: src/%.c build/dirs     $(CC) -MF build/d/$(subst /,.,$(basename $@)) -o $@ $<

 
The pattern rule used for invoking the C compiler has changed. Output files are placed in the build directory, and input files come from the src directory. The substitution rule is used to remove the directory separators from the filename that's used for the dependency files, so that they'll all be placed directly in build.d. If they were allowed to be placed in subdirectories of build.d, we'd have to create those subdirectories manually, which would be a hassle.
 

# Pattern rule for injecting version numbers into files build/%.sed: src/template/% src/Version build/dirs     sed -f src/Version $< > $@

 
Another pattern rule is used to automate injection of the package version number and date into files: Any file X placed in src.template can have its processed version available as build.X/sed (or build/X.sed as a Unix path). The sed extension is just a convenient way of making sure the rule acts on the right files.

build/dirs

Both of the above rules are also configured to depend on the build/dirs rule - which is used to make sure the build directory (and critical subdirectories) exist prior to any attempt to place files in there:
 

build/dirs:     cdir build     cdir build.o     cdir build.d     create build.dirs

 
The file build.dirs is just a dummy file which is used to mark that the rule has been executed.

Explicit dependencies

# Explicit dependency needed for generated file build/VersionNum.sed build/suned.o: build/VersionNum.sed

 
Although most C dependencies are handled automatically via the -MF compiler flag (and the -include makefile directive), some extra help is needed for build.VersionNum/sed because the file won't exist the first time the compiler tries to access it. By adding it as an explicit dependency, we can make sure it gets generated in time (although it does require some discipline on our part to make sure we keep track of which files reference build.VersionNum/sed)

Double-colon rules

# Double-colon rules execute in the order they're listed. So placing this rule # here makes sure that the 'build' folder exists prior to the rule below being # executed. build/pkg-mani:: build/dirs   # Double-colon rules with no pre-requisites always execute. This allows us to # make sure that build/pkg-mani is always up-to-date build/pkg-mani::     src/manigen src.pkg build.pkg-mani

 
Double-colon rules. The manigen program solves the problem of telling make when the contents of a directory have changed, but it leaves us with another problem: We need to make sure manigen is invoked whenever the folder we're monitoring appears in a build rule. The solution for this is double-colon rules, because they have two three very useful properties, which are exploited above:

1. A double-colon rule with no pre-requisites will always execute (whenever it appears in the dependency chain for the current build target(s)). This is the key property which allows us to make sure that manigen is able to do its job.
2. You can define multiple double-colon rules for the same target.
3. Double-colon rules are executed in the order they're listed in the makefile. So by having a rule which depends on build/dirs, followed by the rule that depends on nothing, we can make sure that the build/dirs rule is allowed to create the build folder prior to manigen in the second rule writing its manifest into it.

Of course, we could have just used one build/pkg-mani rule which manually creates the build folder every time it's executed. But the two-rule version is less hacky, and that's kind of the point of this exercise.

Creating the package directory

This is a fairly lengthy rule which does a few different things, but they're all pretty simple.
 

# Create the package dir ready for zipping build/pkg-dir: build/pkg-mani build/!RunImage build/Control.sed build/!Help.sed COPYING # Copy over the static files     x wipe build.pkg ~CFR~V     $(CP) src.pkg build.pkg $(CPOPT) # Populate the RiscPkg folder     cdir build.pkg.RiscPkg     $(CP) build.Control/sed build.pkg.RiscPkg.Control $(CPOPT)     $(CP) COPYING build.pkg.RiscPkg.Copyright $(CPOPT) # Populate the app folder     $(CP) build.!Help/sed build.pkg.$(ROAPP).!Help $(CPOPT)     $(CP) build.!RunImage build.pkg.$(ROAPP).!RunImage $(CPOPT) # Create the dummy file we use to mark the rule as completed     create build.pkg-dir

 
Since there are many situations in which the copy command will not copy, I've wrapped up the right options to use in a variable. Care is taken to specify all the options, even those which are set to the right value by default, just in case the makefile is being used on a system which has things configured in an odd manner.
 

CP = copy CPOPT = A~CF~NQR~S~T~V

 
In this case some of the options are redundant, since this rule completely wipes the destination directory before copying over the new files. But for bigger projects it might make sense to build the directory in a piecemeal fashion, where the extra options are needed.
 
Once the directory is built, the binary.zip rule can produce the resulting zip file:
 

# Binary RiscPkg archive binary.zip: build/pkg-dir     remove binary/zip     dir build.pkg     zip -rqI9 ^.^.binary/zip *     dir ^.^

 
Note that in this case I could have merged the binary.zip and build/pkg-dir rules together, since build/pkg-dir is only used once. And arguably they should be merged together, just in case I decide to test the app by running the version that's in the build.pkg folder, but it then writes out a log file or something that then accidentally gets included in the zip when I invoke the binary.zip rule later on.
 
But, on the other hand, keeping the two rules separate means that it's easy to add a special test rule that copes the contents of build.pkg somewhere else for safe testing of the app. And as mentioned above, for big apps/packages it may also make sense to break down build/pkg-dir into several rules, since wiping the entire directory each time may be a bit inefficient.

In closing

With a setup like the above, it's easy to automate building of packages for applications. Next time, I'll be looking at how to automate publishing of packages - generating package index files, generating the pointer file required for having your packages included in ROOL's index, and techniques for actually uploading the necessary files to your website.