RISC OS Build service can be used with Continuous Integration ('CI') systems to build RISC OS components automatically. The API documentation describes the API that is used by the service. The Build configuration describes configuration files that can be used to describe the RISC OS build process.
This document describes how these can be used together with CI systems. The CI systems that are discussed are:
These build tools can be used with the JSON API,and examples will be given for curl and jq. However, the service is easier to access with the 'robuild-client' tool. Both these methods will be described.
It is assumed that:
- The projects being built are in a git repository.
- The project is stored with non-RISC OS filename encoding, eg '
- A '
.robuild.yaml' file has been created which can build the project.
- There is a passing familiarity with shell scripting on unix systems.
JSON API with curl
Simple build and status
The simplest example of submitting a file
my-source-file to the service with curl would be a command like:
The result is written to the file
/tmp/output. However this assumes that the build was successful - if it is unsuccessful the result will be a 400 response with the output written to the body. The 400 code is silently ignored by the response here.
It is, however, possible to capture the status code with a little a capture and assignment:
One caveat for submitting single files which you should be aware of is that the
to specify the file to submit does not support passing filenames with commas in.
This means that if you are submitting a filename encoded with the RISC OS filetype format you will need
to copy the file before hand.
Extracting more information
The above example suffice for simple submissions to the service, but you may wish to do more than this. The JSON interface allows you to extract more information about the response from the service.
jq' tool can be used to extract information from the output JSON file. For example to extract the messages you might use:
The resulting data, if any, is returned in the '
data' key, and the RISC OS filetype in the 'filetype' as an integer.
The returned data is Base64 encoded, so will need to be decoded before it can be used.
Of course, it is common to want to stop when the build fails. This is simplest if we just check the return code for non-0, and exit the build process at that point.
Handling multiple files
Whilst it's nice to be able to work with a single file, that's commonly not how projects are structured. Commonly there are multiple files that make up the project, and that's where the zip
archives and '
.robuild.yaml' build configuration becomes useful. If you have a repository
containing files, together with a '
.robuild.yaml' file, these can be transferred to the
server as a Zip archive.
Bringing it all together
To script this in a generic way, I put together the fragments in a way that makes it easy to reuse in different cases. This could be in your own scripts or in a build system like Jenkins. The basic submission and data collection script I have used is:
This produces a set of files, in a temporary directory, and a couple of environment variables which can be used to decide what to do with the build.
To decide what to do you might print out the output messages, and report the failures if any:
robuild-client' tool is intended to make it easier to drive the build service through the
websockets interface by removing the need for manually manipulating the JSON files and extracting content,
and to give a more interactive environment to see the output as it happens. The tool works on Linux, macOS
and RISC OS. It should be trivial to port to Windows, but this has not been attempted as yet.
Unlike the manual method above, we must first obtain the build client tool, before we can submit the files to the service. Whilst the tool could be installed in your environment, when used from CI, it's commonly easier to just download it as needed.
The result will be written to the file '
/tmp/build,XXX'. The job has been given a timeout of
60 seconds. This can be useful if you're concerned that it might get into an infinite loop. There is a timeout
on the service, but this is system controlled, so it is better to specify a timeout to what you feel is
The system output and the build output will be written to the terminal, which means that it is not ncessary
to do any more parsing of files.
Although this requires a download of the tool on each invocation, it's simpler and more maintainable in a CI environment than the shell code which did similar operations. Consult the 'robuild-client' repository for more details on the tool.
GitHub can trigger builds when changes are pushed to branches or tags. The definition of what build is triggered is called a 'workflow', and the workflows can trigger multiple actions. The workflows are described in YAML, which is described in the GitHub documentation.
The YAML file is held in the files '
There can be multiple independant workflows which will be run in parallel, differentiated by the filename '
The YAML content looks like this:
- The top level '
name' key allows different workflows to be distinguished from one another.
- The '
on' block defines which git server operations will cause the workflow to be run. In the example, this is on all branches and all pull requests, as this is likely to be the most common set of triggers that will be required.
- The '
jobs' block defines multiple jobs which can be run to perform the build or test. In the example, only one job is present, for the build of a RISC OS component. Some projects may wish to include other types of builds; for example the robuild-client workflow has three distinct builds present, which are serialised (build for Linux, build for RISC OS, create release).
- Within each of the builds, there are other properties, which can refine how the build happens.
- The '
runs-on' key defines the environment on which the steps in this build of the workflow will be run. The robuild-client is built for ubuntu, so this is used in the example.
- The '
steps' block describes the steps that will be run to make the build.
- The '
uses' key allows 'actions' - canned operations - to be performed. There are many actions available to GitHub workflows. The example uses just one of the 'checkout' operations, which checks out the git source into the working directory.
- The '
runs' key allows a shell script to be executed. This is used with the '
name' key, which gives the step a name. The value of the '
runs' element is a list of lines to run. In YAML, this list is indicated by the '
|', and ends when the indentation of the line returns to that of the introducing key. Consult the YAML documentation for more detail (but be prepared for headaches).
Artifacts and version numbering
This configuration can be combined with the scripting used in the earlier examples to give a build of the RISC OS component. To be useful as a tool for building binaries that others may use, though, it is necessary to archive the output into an 'artifact'. In building an distributable component, it is common to want to distribute the results with a version number or other differentiating feature.
Version numbering can be performed in a number of ways, but it is common to have a '
file which contains '
#define' statements to declare the version number. This is the way that
the RISC OS source manages its versions. However, if no version is handled this way, it may be more useful
to just use the commit identifier (aka 'Git SHA'). These operations can be performed with a YAML file that
looks like this:
- The '
outputs' block defines that there will be variables generated by an identified step. These variables can be used in later steps, or in different builds. We use them here to declare what we called the archive that we built, and the version number we determined.
- A new step is created which will create extract the version number from the
VersionNumfile, or use the commit identifier if no version file was found. Other methods might be to extract from a tag, or to use a different file to read the version from.
The magic output '
::set-output ...' indicates the variables that should be set and their values.
- The built output is, here, assumed to be a Zip archive. Zip archives will be created by the build service if the build configuration specifies an artifact path, rather than using the clipboard to copy a file. Filetype &a91 is used for Zip archives.
The final step uses the '
upload-artifact' action to transfer the artifact from the CI server to GitHub.
Artifacts stored by GitHub are only available for a short period. In order to be retained so that they can be used for distribution, it is necessary to create a 'Release' in GitHub. This can be achieved at the end of the build process. Commonly, releases are only created when a block of work has been completed and the author is happy with the results. This can be achieved by using git 'tags' to indicate that a release is required. Using any tag prefixed by a 'v' is a common way to achieve this.
- The '
on' block has been extended to trigger on the creation of tags that start with a '
- The '
build-riscos' block has been elided.
- A new job has been created which follows on from the earlier job, due to the '
- The 'release' job, is conditional with the '
if' key which restricts it to only run when the tag has been pushed which starts with a '
- The binary we created as an artifact is downloaded with the '
- The '
create-release' action is used to create a new release into which we can put the artifact. There is a name, based on the version number we determined earlier, generated for the release. In theory this could be based on the tag name, but then the version number in the built component might not match that of the release. The release is created as a 'draft' so that it must be approved before being public.
upload-release-asset' action is then used to put the content that was downloaded into the release. This must use a magic token (because we don't want any old user uploading a release to your project). The name of the file that will be uploaded, its content type, and where it can be found are given with the '
asset_' prefixed keys.
Once released, it is necessary for the author to confirm the release (because it was a 'draft'). This allows the author a chance to test that what was built is actually suitable for distribution. Releases do not expire by default and will remain available to the public.
The CObey component gives an example of how you might build components and releases automatically. It is not required to follow the pattern given here - there are other ways to achieve this automation which you can use as you wish.
GitLab CI is able to build on commits in much the same way as the GitHub actions. It is a very powerful system, but doesn't have the same 'action' infrastructure as the GitHub. Consequently, a simpler example is supplied here which shows how to use CI with GitLab. There is fuller documentation on the GitLab documentation site.
As with the RISC OS Build service build configuration, and GitHub workflow, the configuration for GitLab CI is YAML. GitLab CI organises the builds into stages, which may contain multiple builds. The builds will happen in parallel, unless a dependency is introduced. Each build may contain multiple scripted steps, which are run in the shell on the target system.
The basic structure of the GitLab CI file is thus:
- The '
stages' block declares which named stages will be built, and the order in which they will be built. If any of the builds within the stages fails, the stage will fail, and the job will be terminated.
riscos' block is a named build. It contains a '
stage' key which declares the stage this build will be executed within.
script' block is a list of shell commands to run. The failure of any of these commands will terminate the build with a failure. As with the GitHub shell commands, these are usually indicated by a '
|' block which introduces lines of commands.
artifacts' block declares which files should be archived as artifacts.
tags' block indicates which environments (which 'runner' in GitLab terms) the code should run on. This will depend on your installation as to which builder is appropriate.