Advanced Code Tests

Before you read about the Advanced Code Test, please be sure to check the Standard Code Tests as these require no coding at all, are very quick to set up and cover the majority of test cases you are likely to want to create.

The Code Test assessment type allows you to write code that checks code a student has written. You can write the code in any language you like provided it can be run from the command line, which covers almost all likely scenarios.

Sample Starter Pack

There is a Starter Pack project that you can add to your account. Click here to install. This project contains examples for all types of auto-graded assessments as well as a Codio Guides cheat sheet.

Test definition

Setting up a Code Test within the Guide editor is very simple. As you can see from the screenshot below, you should specify the following details

  • Name is a short name that describes the test. This name will appear in teacher dashboards, so naming it clearly is important so teachers can see precisely which challenges are successfully met (or not) by students. In many cases, you do not want to see this text appear within the challenge text the student sees. To suppress this text, flip the switch next to the name field.
  • Points is the score given to the student if the code test passes. You can choose any positive numeric value.
  • Command is the command to run to invoke your test. See the section ‘Test code’ for more details.
  • Instructions is the actual text that should be shown to the user, written in Markdown.


Test Code

You can place your test code anywhere you like. One suitable location is to create a .guides/tests folder.

When specifying a command to run, you could specify it in either of these ways (the ~/workspace folder is assumed if you do not specify a full path).

node .guides/tests/mytest.js
node /home/codio/workspace/.guides/tests/mytest.js

This example assumes NodeJS, but you can use any language you like.

Test framework

Codio provides a simple framework for communicating with the Codio Guide. When defining a test, you specify the points that should be awarded for a successful answer.

Success or Failure

To let Codio know whether the test passed the test, your code should simply exit with 0. A Bash script would return with exit(0), NodeJS with process.exit(0) etc.

To indicate failure, exit with a non-zero value.

Partial Score

Codio also provides a way for you to award a partial score rather than the all or nothing approach described above.

If your test was written using a bash script, it is done like this.

curl -s "$CODIO_PARTIAL_POINTS_URL&points=${POINTS}" > /dev/null

A Python script might look like this.

#!/usr/bin/env python
import os, requests, sys

points = 5
url = "{0}&points={1}".format(os.environ['CODIO_PARTIAL_POINTS_URL'], points)
r = requests.get(url)

The score you award should be any value between 0 and the maximum score you specified when defining the assessment in the Codio Guides editor.

Displaying information to the student

You can return text to the user that is shown once the test has concluded. Your test output is captured from stderr and stdout, so for NodeJS, for example, console.log('Well done!!') would work.

For success, you might simply return 'Well done!’. For failure, the more information you provide the better so the student ideally has some clear indication why they got it wrong.

You can return plain text but if you want to format your response text, you can return html. To do so, make sure you enclose your html within <html> </html> tags.

Dashboard score

If your test returns 0, then Codio will give a dashboard score specified in the Points field. If it returns a non-zero value, then a score of 0 is assumed.

Inputs and Outputs to student code

If your assessment requires that inputs are passed into the student code or data should be returned from the student code, then it is entirely up to you how to implement this. You should make it clear how the student should process your test’s inputs and how to return data back to your test.