Page History

...

AWS Account
Terraform Cloud Account
Preconfigured access in ~/.terraformrc . Get the token from https://app.terraform.io by going to Settings → Teams → Team API Token. Generate a new token and create the file ~/.terraformrc
Code Block
credentials "app.terraform.io" { token = "iz5o8MNxgBBPwQ...." }

We need to maintain two workspaces - one for the Fabric Kubernetes cluster and one for the openIDL applications.

To create the workspaces use the tool located in senofi/openidl-devops:

Go to openidl-devops/aws-infrastructure/environments/<env-folder>/terraform-cloud and run

#

Step

1

Setup

Check out repository senofi/openidl-devops
Create a new folder under openidl-devops/aws-infrastructure/environments/ by copying the sample folder openidl-devops/aws-infrastructure/environments/sample-env

Note
Make sure there are no other credentials in the `~/.terraform/`

folder

folder (if it exists) as they will take precedence over the ones in file ~/.terraformrc

2

Create IAM User & Role

Pull the AWS credentials from AWS Console for the AWS account you have access to. They could be set in the console or configured:
export AWS_ACCESS_KEY_ID="ASIAWRO3..." export AWS_SECRET_ACCESS_KEY="6gLd..." export AWS_SESSION_TOKEN="IQoJpZ2l..." # only used for live session key
The AWS IAM user needs to have access to IAM to create roles and other users.
Go to openidl-devops/aws-infrastructure/environments/<env-folder> as copied in the previous section
Configure openidl-devops/aws-infrastructure/environments/<env-folder>/org-vars.yaml
1. Fill in the IAM AWS access and secret keys under section iam of the YAML file
2. Configure the org ID and the environment ID (dev, test or prod)
Go to <env-folder>/iam and run terragrunt plan
After a review apply the changes with terragrunt apply

The script creates:

IAM role (for used by the terraform user)
IAM user (terraform user)

3

Create Ops Kubernetes Cluster

Register manually a new SSH key pair in AWS by going to EC2 → Key pairs (RSA, pem file). Create a new key with a name awx-target Keep the private key in the environments folder or anywhere on the file system you prefer
Go to the Terraform Cloud workspace that was just created in the previous section and go to the States tab. Open the top state in the list and find outputs and copy access_key and secret_key values that will be used for the next step
Go to <env-folder>/k8s-cluster and run terragrunt planThe previous step should fail but it should have created a new workspace in Terraform Cloud - e.g. devnet-d3-k8s-cluster
Make sure the AWS variables are set in Terraform Cloud as Terraform variables under the workspace in the previous step → Variables tab org-vars.yaml under terraform: property
1. aws_access_key = terraform user’s access key ID
2. aws_secret_key = terraform user’s secret access key
3. region = us-east-2 or any other region you prefer
4. aws_role_arn = terraform role ARN
5. aws_external_id = terraform
Run again terragrunt plan
Review and if things look ok run terragrunt apply
Acknowledge the run with yes in the prompt

The script creates:

Kubernetes cluster
PostgreSQL DB for Ansible Tower (AWX)
VPC, network

4

Import the Kubernetes Cluster connection config

Make sure you have an AWS profile or key/secret/session environment variables set onto the terminal sessionprofile set in your ~/.aws/config and ~/.aws/credentials

Code Block

title	~/.aws/config

[profile tf-user]
region = us-east-2
external_id = terraform

[profile tf-role]
external_id = terraform
source_profile = tf-user
role_arn = arn:aws:iam::<aws-account-number>:role/tf_automation
region = us-east-2

Code Block

title	~/.aws/credentials

[tf-user]
aws_access_key_id = AKI...
aws_secret_access_key = r3AB...

Find the name of the Kubernetes cluster and update the local config with it

Code Block
export AWS_PROFILE=tf-role aws eks update-kubeconfig --name ops-k8s

5

Install Nginx

Install Nginx Ingress controller

Code Block
kubectl create ns ingress-nginx helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm install -n ingress-nginx lb ingress-nginx/ingress-nginx

Info
It is possible that the nginx LB will not be assigned DNS and IP due to the security group for the cluster and the nodes tagged with the same annotation. To fix that find the security group for the nodes (e.g. ops-k8s-node) and remove the owned tag.

6

Install Jenkins

Use the helm chart for installing Jenkins onto the Kubernetes cluster created above.

Code Block
cd <devops-repo>/jenkins kubectl create ns jenkins helm repo add jenkins https://charts.jenkins.io helm upgrade --install -n jenkins jenkins jenkins/jenkins --values values.yaml

Wait for Jenkins to start up.

To view the Jenkins admin password:

Code Block
kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo

Set up a cloud-provisioned Jenkins node as defined in the Kubernetes plugin config in Jenkins.

7

Install Ansible Tower (AWX)

Create the AWX DB by connecting to the RDS PostgreSQL instance created via Terraform.

Create an SSH Tunnel. Lookup the RDS DB DNS and the EC2 instance that is the AWX target public DNS and replace them in the command line template:

Code Block
ssh -i <env-folder>/awx-target.pem -N -L 5432:ops-tools-db.<instance-id>.us-east-2.rds.amazonaws.com:5432 ubuntu@<awx-target-ec2>.us-east-2.compute.amazonaws.com -vv

Connect with DBeaver (or another PostgreSQL client) on localhost port 5432 and run the following SQL after replacing <pass> with an actual password (as defined under environments/<env>/org-vars.yaml )
Code Block
create database awx; create user awxuser with encrypted password '<pass>'; grant all privileges on database awx to awxuser;
Configure the kustomize Kustomize script awx-custom.yaml by replacing the DB settings in awx-operator folder under openidl-devops Git repository.

Install AWX with the Kustomize scriptcommand.

Code Block
cd awx-operator helm repo add awx-operator https://ansible.github.io/awx-operator/

# clone the gitops repo and go to awx-operator kustomize cd awx-operator kustomize build . | kubectl apply -f


kustomize build . | kubectl apply -f -

Watch for the script failing and if it does run it again (timing issue due to the creation of the AWX RBAC)

8

Update DNS record (optional)

Go to the AWS Account → Route53
Create a new Hosted Zone (e.g. d1.test.senofiopenidl-org-test.net)
Under the new hosted zone create a new entry of type A with an Alias for the Kubernetes cluster (e.g. ops.d1.test.senofiopenidl-org-test.net) to point to a Classic Load Balancer

Now Jenkins and AWX should be available via http://ops.d1.test.senofiopenidl-org-test.net/ and http://ops.d1.test.senofi.net/jenkins.

9

Terraform Cloud workspaces

Code Block
terragrunt plan

If everything looks ok, execute terragrunt apply. This should create two workspaces and a var set in Terraform Cloud.

Create a new KMS key (symetric, encrypt/decrypt) in the AWS console. The name is not important but use a meaningful name that will associate it with this environment. Use it to populate the property in the next step

Go to openidl-devops/automation/terraform-cloud and update configuration.properties Make sure that the varset name

Create SSH keys

Code Block
ssh-keygen -t rsa -f app_eks_worker_nodes_ssh_key.pem ssh-keygen -t rsa -f blk_eks_worker_nodes_ssh_key.pem ssh-keygen -t rsa -f bastion_ssh_key.pem

Populate the variable set by executing the following command in openidl-devops/automation/terraform-cloud

Code Block
pip install -r requirements.txt python populate-variable-set.py

Copy the contents of the public keys and populate them in Terraform Cloud UI under Variable Sets → <the newly created varset>

10

Configure Jenkins

Set Jenkins node label ‘openidl’ in Kubernetes Cloud by going to Manage Jenkins → Manage Nodes and Clouds → Configure Clouds. Make sure that under Pod Template details the labels field contains the value ‘openidl’.
Also, remove the prepopulated ‘sleep’ command if it is set on the pod template:
Create the Terraform Job Template
1. Terraform Token Secret - Login to Jenkins go to Manage Jenkins → Manage Credentials → Stores scoped to Jenkins (Jenkins) → Global Credentials (unrestricted) → Add credentials
  Choose Kind as secret text, enter secret text like Token in “secret” field and name the secret ID as unique since it will be used in pipeline code.
2. Git Credentials - Add a new credential
Terraform Job
1. Go to Jenkins → New Item. Use a name such as Terraform Job
2. Select job type as PIPELINE and proceed.
3. Select Definition as Pipeline Script from SCM
4. Select SCM as Git
5. Key in the Infrastructure code repository (openidl-gitops) URL.
6. Select the Git credential created above
7. Specify the relevant branch “refs/heads/<branch-name>”.
8. Set script path to jenkins-jobs/jenkinsfile-tf

11

Run Terraform Job

Run the Jenkins Terraform Job
Open the console log for the job. Once the job asks for an input accept and choose the apply option
The job runs a second plan into the Kubernetes workspace in Terraform Cloud. When asked - accept and apply the changes
Go to the AWS Console and find EKS (Elastic Kubernetes Service). Choose the blk cluster and go to Add-Ons. Find the EBS plugin and add it to the list. The plugin makes sure volumes could be created in Kubernetes

AWX Setup

Prerequisites:

AWX is installed and operational
AWS infrastructure is provisioned
Configuration is done and available at a private git repository
Credentials information is defined and available
- aws iam user
- git private repo deploy key
- bastion ssh private key
- hds db access
- fabric console user and password
Bastion machine (gateway) host address is available

Steps:

...

Step

...

Notes

...

User

...

Checklist

...

Create new Organization

...

Create new organization with the org_id

...

admin

...

Setup new org AWX user

...

Create new user named with the org name, assing admin permissions to the created organization above

...

admin

...

Setup org project

...

Create new project named with the org name, use openidl ansible git url and the appropriate branch.

...

org user

...

Setup inventory

...

Create new Inventory named with the org name (bastion-org_id).

Add host using the bastion machine addrress

Add group named ansible_provisioners

Add the bastion host to the group

...

org user

...

Create Credentials

...

Create the credential types as specified below (see credentails table)

...

admin

...

Create AWX job tempaltes

...

Create the AWX job tempaltes as specified below (see AWX job tempaltes table)

...

org user

Credentials:

...

Credential

...

Description

...

Definition/Type

...

Checklist

...

aws-git-actions

...

A AWS credential used to access AWS APIs. The IAM user is created during the AWS provisioning step. This user usually should have access to AWS resources and the provisioned k8s clusters (hlf and applications k8s). The user is usually named suffixed with git-actionsadmin. External AWS id is usually git-actions.

The user is used by the playbooks to perform the deployment and setup actions.

Code Block

fields:
  - id: aws_access_key
    type: string
    label: aws_access_key
    secret: true
    help_text: AWS IAM user access key for aws
  - id: aws_secret_key
    type: string
    label: aws_secret_key
    secret: true
    help_text: AWS IAM user secret key for aws
  - id: aws_external_id
    type: string
    label: aws_external_id
  - id: aws_assume_role_arn
    type: string
    label: AWS IAM user role to assume
required:
  - aws.access_key
  - aws.secret_key
  - aws.external_id
  - aws.assume_role_arn

Code Block
extra_vars: aws_access_key: '{{ aws_access_key }}' aws_secret_key: '{{ aws_secret_key }}' aws_external_id: '{{ aws_external_id }}' aws_assume_role_arn: '{{ aws_assume_role_arn }}'

...

git-config

...

Git credentials (used to pull configuration from the private repository)

Code Block
fields: - id: sshkey type: string label: Base64 encoded deploy private key string secret: true - id: repourl type: string label: GIT repo URL - id: repobranch type: string label: Git repo branch

Code Block
extra_vars: ssh_key: '{{ sshkey }}' git_configs_repo_url: '{{ repourl }}' git_configs_repo_branch: '{{ repobranch }}'

...

bastion

...

Bastion Machine SSH credential.

This machine is boostrapped during the AWS infrastructucture provisioning step. It is used as a remote agent for the ansible playbooks. It is the entry point (gateway) to access the AWS infrastructure in order to setup and deploy the network.

...

Machine - an existing standard credenatial in AWX

...

hds-access

...

Access information for application HDS DB.

This credetnails is injected by the playbooks to configure the openIDL applications for access to the local carrier HDS database. The ansible playbooks don’t use it to establish connection to the HDS and perform operations.

Code Block

fields:
  - id: hds_host
    type: string
    label: HDS host
    help_text: HDS host address
  - id: hds_port
    type: string
    label: hds_port
    help_text: HDS port
  - id: hds_username
    type: string
    label: hds_username
    secret: true
  - id: hds_password
    type: string
    label: hds_password
    secret: true
  - id: hds_dbname
    type: string
    label: hds_dbname
required:
  - hds_host
  - hds_port
  - hds_username
  - hds_password
  - hds_dbname

Code Block
extra_vars: hds_host: '{{ hds_host }}' hds_port: '{{ hds_port }}' hds_dbname: '{{ hds_dbname }}' hds_password: '{{ hds_password }}' hds_username: '{{ hds_username }}'

...

fabric-console

...

Fabric Operator Console access default user / password.

Used by the playbooks to inject default user and password for the fabric consle deployment. Make sure the generate a strong password as it will secure properly the access to the node HLF managed.

The playbooks also use this credentail to connect to the consle for the purpose of performing operations on the HLF nodes.

Take a note of that credential as the provided user and password will be requried to login to the fabric operator console.

Code Block

fields:
  - id: console_username
    type: string
    label: console_username
    help_text: Fabric Operator Console Username
  - id: console_password
    type: string
    label: console_password
    secret: true
    help_text: Fabric Operator Console Password
required:
  - console_username
  - console_password

Code Block
extra_vars: console_password: '{{ console_password }}' console_username: '{{ console_username }}'

AWX Job Templates:

Playbook

Template Name

Credential

Description

Checklist

ansible/environment-setup.yaml

<env_id>-<org_id>-environment-setup

aws-git-actions

bastion

git-config

Install open source tools on the bastion host.

Setup the access to the cloud APIs

ansible/deploy-fabric-ingress.yml

<env_id>-<org_id>-deploy-fabric-ingress

aws-git-actions

bastion

git-config

Deploy Ingress controllers (classes) and cloud load balancers for the HLF k8s cluster

ansible/deploy-vault.yml

<env_id>-<org_id>-deploy-vault

aws-git-actions

bastion

git-config

Deploy Vault raft cluster for storing HLF identities (application and HLF nodes admins)

ansible/deploy-fabric-operator.yml

<env_id>-<org_id>-deploy-fabric-operator

aws-git-actions

bastion

git-config

Deploy HLF fabric operator

ansible/deploy-fabric-console.yml

<env_id>-<org_id>-deploy-fabric-console

aws-git-actions

bastion

git-config

fabric-console

Deploy HLF operator console

ansible/deploy-openidl-app-identities.yml

<env_id>-<org_id>-deploy-app-identities

aws-git-actions

bastion

git-config

fabric-console

Registers and enrolls the openidl application identities used to transact on the openidl fabric network

ansible/deploy-openidl-app-ingress.yml

<env_id>-<org_id>-deploy-app-ingress

aws-git-actions

bastion

git-config

Deploys the application ingress controller and class. Creates the applications load balancers for the applications k8s cluster.

ansible/deploy-mongodb.yml

<env_id>-<org_id>-deploy-mongodb

aws-git-actions

bastion

git-config

Deploys mongoDB as application database

ansible/deploy-openidl-app-config.yaml

<env_id>-<org_id>-deploy-app-config

aws-git-actions

bastion

git-config

fabric-console

hds-access (carrier node)

Generates and deploys the application config as secrets

ansible/deploy-openidl-app.yaml

<env_id>-<org_id>-deploy-app

aws-git-actions

bastion

git-config

Deploys the openidl applicationsopenidl-org-test.net/jenkins.

Space shortcuts

Page tree

Versions Compared

Old Version 9

New Version Current

Key

AWX Setup

Prerequisites:

Steps:

Credentials:

AWX Job Templates: