Preparing your Azure environment

MettleCI ships with a sample Azure DevOps repository which includes a number of example pipelines definitions as well as a shell script which demonstrates how to use the Azure command line to establish the requires assets in your Azure environment. This page describes the contents of that file.

Collecting your required information

1. Connectivity

   CODE

   export ADO_USERNAME=myuser.name@mycorp.com                      # Your Azure DevOps username
   export ADO_PASSWORD=MyPassword                                  # Your Azure DevOps password
   export ADO_ORGANIZATION_URL=https://dev.azure.com/mettleci      # Your Azure DevOps organization's URL
   export ADO_PROJECT=MyProject                                    # Your Azure DevOps project name (this project doesn't need to pre-exist)

2. Git repository references

   CODE

   export ADO_REPOSITORY_DATASTAGE=MyAzureDataStageRepo
   export ADO_GIT_SOURCE_URL_DATASTAGE=https://dev.azure.com/myorg/my-template-project/_git/MyAzureDataStageRepo
   export ADO_REPOSITORY_COMPLIANCE=MyAzureComplianceRepo
   export ADO_GIT_SOURCE_URL_COMPLIANCE=https://dev.azure.com/myorg/my-template-project/_git/MyAzureComplianceRepo

3. Variable groups

   CODE

   export ADO_VARGROUP_NONPROD=MyProject_NONPROD                   # The name of the variable group for variables describing your non-production environment
   export ADO_VARGROUP_PROD=MyProject_PROD                         # The name of the variable group for variables describing your production environment

Create your Azure Project and configure your CLI environment

1. Connect to your Azure DevOps instance by opening a browser-based interactive authentication session:

   CODE

   az login

2. Create an Azure DevOps project to host your DataStage assets and Azure build pipelines:

   CODE

   az devops project create \
       --name $ADO_PROJECT \
       --description 'MettleCI-enabled DataStage project' \
       --org $ADO_ORGANIZATION_URL \
       --source-control git \
       --visibility private

3. Set the default organisation and project for the Azure CLI environment:

   CODE

   az devops configure \
       --defaults \
       organization=$ADO_ORGANIZATION_URL \
       project=$ADO_PROJECT

Agents and Agent Pools

Agent pools can be created easily using the Azure DevOps UI. Neither the number of pools you create, nor the names you give them, are relevant to your MettleCI-enabled pipelines as jobs are automatically assigned to agents by Azure DevOps which matches the demands of each of your pipelines' steps with the capabilities advertised by your agents.

The definition of Agents requires you to install one or more self-hosted Azure agents on a suitably equipped host (see https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/windows-agent?view=azure-devops) and associate the agent(s) with a relevant agent pool.

1. List your agent pools:

   CODE

   az pipelines pool list \
       --organization $ADO_ORGANIZATION_URL \
       --output table

2. Select (or create) your preferred agent pool and assign its ID to $ADO_AGENT_POOL_id (for subsequent use):

   CODE

   export ADO_AGENT_POOL_id=$(az pipelines pool list \
       --organization $ADO_ORGANIZATION_URL \
       --query '[?name == `MyFavouriteAgentPool`].id' \
       --output tsv)

3. List your agent pools' constituent agents:

   CODE

   az pipelines agent list \
       --organization $ADO_ORGANIZATION_URL \
       --pool-id $ADO_AGENT_POOL_id \
       --output table

Environments

The creation of Deployment Environments is not currently supported by the Azure CLI. Environments are created by the supplied pipelines as they are references. i.e., if you try and run a MettleCI deployment to an environment called MyQualityAssurance then environment of that name will be automatically created. Once a deploy environment has been created (either manually using the Azure UI or automatically by reference from a pipeline) you can then configure its Approvals and checks settings to restrict deployment to that environment as required.

Repositories

1. Create DataStage repository:

   CODE

   export ADO_REPO_DATASTAGE_id=`az repos create \
     --name $ADO_REPOSITORY_DATASTAGE \
     --org $ADO_ORGANIZATION_URL \
     --project $ADO_PROJECT \
     --query 'id' `

2. Import DataStage repository from source repository ($ADO_GIT_SOURCE_URL_DATASTAGE):

   CODE

   az repos import create \
     --git-source-url $ADO_GIT_SOURCE_URL_DATASTAGE \
     --org $ADO_ORGANIZATION_URL \
     --project $ADO_PROJECT \
     --repository $ADO_REPOSITORY_DATASTAGE \
     --requires-authorization

3. Create Compliance repository:

   CODE

   export ADO_REPO_COMPLIANCE_id=`az repos create \
     --name $ADO_REPOSITORY_COMPLIANCE \
     --org $ADO_ORGANIZATION_URL \
     --project $ADO_PROJECT \
     --query 'id'`

4. Create Compliance repository & import from source repository ($ADO_GIT_SOURCE_URL_COMPLIANCE):

   CODE

   az repos import create \
     --git-source-url $ADO_GIT_SOURCE_URL_COMPLIANCE \
     --org $ADO_ORGANIZATION_URL \
     --project $ADO_PROJECT \
     --repository $ADO_REPOSITORY_COMPLIANCE \
     --requires-authorization

Variable Groups

This example creates two variable groups: one containing all the variables required to specify the details of your non-production environment a similar one for your production environment. You’ll need one of these variable groups for each of your distinct DataStage environments. Each variable group also makes use of passwords stored secret variables.

# Create Non-Production Variable Group
export ADO_VARGROUP_NONPROD_id=`az pipelines variable-group create \
  --name $ADO_VARGROUP_NONPROD \
  --variable \
        AgentMettleCmd=C:\MettleCI\cli\mettleci.cmd \
        AgentMettleHome=C:\MettleCI \
        EngineUnitTestBaseDir=/opt/dm/mci \
        IISDomainName=MyNonProductionServicesTier.MyOrganization.com:59445 \
        IISEngineName=MyNonProductionEngineTier.MyOrganization.com \
        IISProjectTemplateDir=/opt/IBM/InformationServer/Server/Template \
        IISUsername=isadmin \
        IISVersion=11.7 \
        MCIUsername=mciworkb \
  --authorize true \
  --description 'Non-production pipeline variables' \
  --organization $ADO_ORGANIZATION_URL \
  --project $ADO_PROJECT  \
  --query 'id' `

# IISPassword secret variable
az pipelines variable-group variable create \
    --group-id $ADO_VARGROUP_NONPROD_id \
    --org $ADO_ORGANIZATION_URL \
    --project $ADO_PROJECT \
    --secret true \
    --name IISPassword \
    --value MyPassword

# MCIPassword secret variable
az pipelines variable-group variable create \
    --group-id $ADO_VARGROUP_NONPROD_id \
    --org $ADO_ORGANIZATION_URL \
    --project $ADO_PROJECT \
    --secret true \
    --name MCIPassword \
    --value MyPassword

# Create Production Variable Group
export ADO_VARGROUP_PROD_id=`az pipelines variable-group create \
  --name $ADO_VARGROUP_PROD \
  --variable \
        AgentMettleCmd=C:\MettleCI\cli\mettleci.cmd \
        AgentMettleHome=C:\MettleCI \
        EngineUnitTestBaseDir=/opt/dm/mci \
        IISDomainName=MyProductionServicesTier.MyOrganization.com:59445 \
        IISEngineName=MyProductionEngineTier.MyOrganization.com \
        IISProjectTemplateDir=/opt/IBM/InformationServer/Server/Template \
        IISUsername=isadmin \
        IISVersion=11.7 \
        MCIUsername=mciworkb \
  --authorize true \
  --description 'Production pipeline variables' \
  --organization $ADO_ORGANIZATION_URL \
  --project $ADO_PROJECT  \
  --query 'id' `

# IISPassword secret variable
az pipelines variable-group variable create \
    --group-id $ADO_VARGROUP_PROD_id \
    --org $ADO_ORGANIZATION_URL \
    --project $ADO_PROJECT \
    --secret true \
    --name IISPassword \
    --value MyPassword

# MCIPassword secret variable
az pipelines variable-group variable create \
    --group-id $ADO_VARGROUP_PROD_id \
    --org $ADO_ORGANIZATION_URL \
    --project $ADO_PROJECT \
    --secret true \
    --name MCIPassword \
    --value MyPassword

Pipelines

1. Create a DevOps Continuous Integration pipeline:

   CODE

   az pipelines create \
       --org $ADO_ORGANIZATION_URL \
       --project $ADO_PROJECT \
       --name MyDataStage-DevOps-CI \
       --description 'DevOps CI' \
       --repository $ADO_REPOSITORY_DATASTAGE \
       --branch master \
       --yml-path /devops-ci.yml \
       --repository-type tfsgit

2. Create a Hot Fix Continuous Integration pipeline:

   CODE

   az pipelines create \
       --org $ADO_ORGANIZATION_URL \
       --project $ADO_PROJECT \
       --name MyDataStage-HotFix-CI \
       --description 'HotFix CI' \
       --repository $ADO_REPOSITORY_DATASTAGE \
       --branch master \
       --yml-path /hotfix-ci.yml \
       --repository-type tfsgit

3. Create a Hot Fix Deploy pipeline:

   CODE

   az pipelines create \
       --org $ADO_ORGANIZATION_URL \
       --project $ADO_PROJECT \
       --name MyDataStage-HotFix-Deploy \
       --description 'HotFix Deploy' \
       --repository $ADO_REPOSITORY_DATASTAGE \
       --branch master \
       --yml-path /hotfix-deploy.yml \
       --repository-type tfsgit

Next Steps

1. Register your DataStage development project in the MettleCI Workbench referencing your Azure DevOps DataStage and Compliance repositories.

2. Test your Compliance connection by running compliance using MettleCI Workbench.

3. Commit a DataStage asset to your new DataStage repository using MettleCI Workbench. This should trigger your new MyDataStage-DevOps-CI pipeline.