Automating project creation with the GitHub API

The MettleCI GitHub DataStage repository templates require the specification of GitHub Environments and Secrets. Some DataStage organisations have multiple development teams working on multiple DataStage development projects, each of which would typically require a dedicated GitHub repository. Creating all of these artefacts manually for a large number of DataStage delivery streams could be very onerous and error prone. A better solution might be to automate this process using a combination of GitHub APIs to provide calls that can minimise the effort of creating multiple GitHub Repositories, Environments and Secrets.

Guides

Prerequisites

To automate the required steps in GitHub, you will require…

The GitHub CLI, which provides a wide range of methods for manipulating the GitHub environment. Refer to GitHub’s documentation for https://cli.github.com/manual/installation.
A user with permissions to create repositories, environments and secrets
A Personal Access Token with suitable permissions

Creating a Personal Access Token

In the top right corner of the browser, open the dropdown menu against the user avatar and select Settings. Expand Developer Settings (at the bottom of the list).

Click on Generate new token, and select:

repo
read:org (under admin:org)
project

Copy the token (referred to hereafter as <ACCESS_TOKEN>).

Authentication against a GitHub Account

Before performing the required actions, we need to authenticate against the GitHub account, using the Access Token.

CODE

gh auth login --with-token <<EOF
<ACCESS_TOKEN>
EOF

Alternatively, set the environment variable GITHUB_TOKEN with the value of <ACCESS_TOKEN>.

Create Repository

When creating the GitHub Repository, there are a number of options, depending on whether you create the remote repository first and add files to the local clone, or flesh out the directory structure first and create the Repository from it.

To create the remote repository first:

CODE

gh repo create <ORGANISATION_NAME>/<REPOSITORY_NAME> --private -clone

Create Environment

Creating an Environment is currently not supported by the GitHub CLI, but can be achieved using the REST API.

CODE

curl -X PUT \
-H 'Authorization: Bearer <ACCESS_TOKEN>' \
-H 'Accept: application/vnd.github.v3+json' \
https://api.github.com/repos/<ORGANISATION_NAME>/<REPOSITORY_NAME>/environments/<ENVIRONMENT_NAME> \
-d '{"reviewers":[{"type":"User","id":<USER_ID>}]}'

For Environments that do not require approval for GitHub Actions to interact with, such as Continuous Integration (CI) environments, line 5 is not required.

To discover the value of <USER_ID> for a GitHub account, navigate to Settings under the account dropdown, then select Emails. The email account created for web-based Git operation is in the format <USER_ID>_<USER_NAME>@users.noreply.github.com

Create Secret

Once the Environment has been created under the Repository, secrets can be added to it. In the case of GitHub, the large majority of values required by the MettleCI GitHub actions are defined within the action code itself, but passwords are added as secrets to the Environment.

CODE

gh secret set <SECRET_NAME> --repo <ORGANISATION_NAME>/<REPOSITORY_NAME> --env <ENVIRONMENT_NAME> <<EOF
<SECRET_VALUE>
EOF