Creating and Preparing a Git Repository for DataStage Assets

A Git repository (or ‘repo’) is a filesystem folder containing a .git folder with a software layer (’Git') managing the files within that folder. This repository retains a history of all changes made to files over time. A Git repository comes in two (often related) types: Local & Remote.

Local repository: A directory on your local filesystem - in this case on your DataStage Engine. When you git commit your code, a version/snapshot is created in your local repository which you then git push to a remote repository.
Remote repository: A remote repository generally lies somewhere outside your system, on a remote machine, and enables collaborative working between multiple contributors. The Remote repository is where developers share their code and acts as the input to Integration testing.

To enable you to get up and running with MettleCI’s Git integration with the minimum fuss MettleCI ships with a zip file containing a local Git repository which provides a template repository structure as well as an example build pipeline definition for your selected CI/CD build tool. This local repository should be unzipped to a filesystem, connected to a remote Git server and pushed to a remote repository which is the target repository to where subsequent MettleCI Workbench Git Commits will be pushed. This process, which is described on this page, can be performed either from any environment with access to git and unzip commands, including Windows, Linux, and MacOS. The command examples used on this page are from Unix/MacOS, but the Windows command line is almost identical: Git commands are identical, but you’ll need to use dir in place of ls.

Create a Blank Remote Git Repository

Start by creating an empty remote Git repository in your Git system of choice and note a reference to it. Use a repository name that will allow you to easily associate it with the DataStage project it is used to represent. See our notes on Organizing your DataStage Solution Repositories.

Here’s an example where we create a remote repository on Atlassian Bitbucket Cloud …

…and copy a reference to it to our clipboard:

Note in this case that we only want the URL and NOT the git clone command.

To create the repository see the relevant instructions for your Git server:

Create a repository in Bitbucket
Create a repository in GitLab
Create a repository in Azure DevOps
Create a repository in GitHub
… or search the web for other Git systems

Install a local Git command line client

Although this it not required for MettleCI it is (using the method below) required to establish a baseline Git repository for use by MettleCI.

See the available downloads and instructions on the Git website. Test it’s been installed correctly:

BASH

$> git
usage: git [--version] [--help] [-C <path>] [-c <name>=<value>]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           [--super-prefix=<path>] [--config-env=<name>=<envvar>]
           <command> [<args>]
etc.

You’ll also need to configure your git client as specified in the Git manual:

BASH

$> git config --global user.email "your.name@example.com"
$> git config --global user.name "Your Name"

Unzip the MettleCI repository template ZIP file

From a command line in the environment where you’ve chosen to perform this operation enter the following commands:

BASH

# Verify you have the repository template file (the filename may be different to this) 

$> ls -l
total 40
-rw-r--r--@ 1 johnmckeever  staff    20K 10 Feb 17:38 mettleci-repository-template.zip

# Unzip it

$> unzip mettleci-repository-template.zip
Archive:  mettleci-repository-template.zip
   creating: mettleci-template/
 extracting: mettleci-template/var.prod
   creating: mettleci-template/filesystem/
  inflating: mettleci-template/filesystem/deploy.sh
<etc.>
 extracting: mettleci-template/var.qa
 extracting: mettleci-template/var.ci
  inflating: mettleci-template/.gitattributes
$> ls -l
total 40
-rw-r--r--@  1 johnmckeever  staff    20K 10 Feb 17:38 mettleci-repository-template.zip
drwxrwxr-x@ 13 johnmckeever  staff   416B 30 Oct  2020 mettleci-template

# Verify its contents

$> cd mettleci-template
$> ls -al
total 48
drwxrwxr-x@ 13 johnmckeever  staff    416 30 Oct  2020 .
drwxr-xr-x   4 johnmckeever  staff    128 10 Feb 17:40 ..
drwxrwxr-x@ 10 johnmckeever  staff    320 30 Oct  2020 .git
-rw-rw-r--@  1 johnmckeever  staff     70 30 Oct  2020 .gitattributes
-rw-rw-r--@  1 johnmckeever  staff  15863 30 Oct  2020 Jenkinsfile
-rw-rw-r--@  1 johnmckeever  staff    704 30 Oct  2020 README.md
drwxrwxr-x@  5 johnmckeever  staff    160 30 Oct  2020 datastage
drwxrwxr-x@  3 johnmckeever  staff     96 30 Oct  2020 filesystem
drwxrwxr-x@  3 johnmckeever  staff     96 30 Oct  2020 unittest
-rw-rw-r--@  1 johnmckeever  staff      0 30 Oct  2020 var.ci
-rw-rw-r--@  1 johnmckeever  staff      0 30 Oct  2020 var.perf
-rw-rw-r--@  1 johnmckeever  staff      0 30 Oct  2020 var.prod
-rw-rw-r--@  1 johnmckeever  staff      0 30 Oct  2020 var.qa
$>

Attach the local repository to the remote repository

Again, from a command line in the environment where you’ve chosen to perform this operation enter the following commands:

BASH

$> pwd
~/mettleci-template

# Re-define it as a local clone of your newly-create remote repository.
# Use the remote reference to your empty DataStage repository provided by your Git system

$> git remote add origin https://myusername@bitbucket.org/myusername/your-project-repo.git
$> git status
On branch master

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	.gitattributes
	Jenkinsfile
	README.md
	datastage/
	filesystem/
	unittest/
	var.ci
	var.perf
	var.prod
	var.qa

nothing added to commit but untracked files present (use "git add" to track)

To verify that you’ve done this successfully you can ask Git to tell you the (newly-configured) remote URL for your local Git repository:

BASH

$> git config --get remote.origin.url
myusername@bitbucket.org/myusername/your-project-repo.git

Note that this will work even if you are not connected to a network that can reach the remote repo. If you want to (more usefully) get a more verbose output, and validate that you are on a network that can reach the remote repository, and that you have the required authentication (e.g. keys) in place, then use the following command:

BASH

$> git remote show origin
* remote origin
  Fetch URL: myusername@bitbucket.org/myusername/your-project-repo.git
  Push  URL: myusername@bitbucket.org/myusername/your-project-repo.git
  HEAD branch: master
  Remote branches:
    master tracked
  Local branches configured for 'git pull':
    master merges with remote master
  Local refs configured for 'git push':
    master pushes to master (up to date)

Note also that if you need to subsequently re-point your repository to a different remote repository for any reason (e.g. you make a mistake, or your situation changes) you can use the git remote set-url command. This is only provided here by way of example - you shouldn't need to run this under normal circumstances.

BASH

# Not normally required
$ git remote set-url origin https://myusername@bitbucket.org/myusername/your-project-repo.git

Add the supplied repository template files to the Git staging area

BASH

$> git add --all .

# Verify your files are added and ready to commit

$> git status
On branch master

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
	new file:   .gitattributes
	new file:   Jenkinsfile
	new file:   README.md
	new file:   datastage/Parameter Sets/.placeholder
	new file:   datastage/cleanup.sh
	new file:   datastage/deploy.sh
	new file:   filesystem/deploy.sh
	new file:   unittest/.placeholder
	new file:   var.ci
	new file:   var.perf
	new file:   var.prod
	new file:   var.qa

Commit the template files you just added

Use any commit message you like.

BASH

$> git commit -am "initial commit of repository template"
[master (root-commit) 89e8152] initial commit to remote, default rules
 12 files changed, 317 insertions(+)
 create mode 100644 .gitattributes
 create mode 100644 Jenkinsfile
 create mode 100644 README.md
 create mode 100644 datastage/Parameter Sets/.placeholder
 create mode 100755 datastage/cleanup.sh
 create mode 100755 datastage/deploy.sh
 create mode 100755 filesystem/deploy.sh
 create mode 100644 unittest/.placeholder
 create mode 100644 var.ci
 create mode 100644 var.perf
 create mode 100644 var.prod
 create mode 100644 var.qa

Push those commits to the new remote Git repository

BASH

$> git push -u origin master
Password for 'https://myusername@bitbucket.org':
Counting objects: 24, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (23/23), done.
Writing objects: 100% (24/24), 20.74 KiB | 0 bytes/s, done.
Total 24 (delta 1), reused 0 (delta 0)
To https://myusername@bitbucket.org/myusername/your-project-repo.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.

# Verify the status of your local repository

$> git show-ref
2b95553dc7572c58e12be36c4f59a52eaf61af8b refs/heads/master
2b95553dc7572c58e12be36c4f59a52eaf61af8b refs/remotes/origin/master

Optional - Remove the now-redundant local Git repository

BASH

$> cd ..
$> pwd
~/mettleci-template
$> rm -rf mettleci-template
$>