Install openIDL Node on AWS

Prerequisites

1. AWS Account

...

1. The user should have access to IAM and permissions to create a new users and roles. The user will be only used to create a Terraform user and role with the following policy contents.

\{
    "Version": "2012-10-17",
    "Statement": \[
        \{
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": \[
                "sns:*",
                "cloudformation:ListStacks",
                "application-autoscaling:*",
                "s3:*",
                "logs:*",
                "cloudtrail:*",
                "dynamodb:*",
                "sts:*",
                "elasticloadbalancing:*",
                "autoscaling:*",
                "iam:*",
                "route53domains:*",
                "secretsmanager:*",
                "cloudwatch:*",
                "ses:*",
                "kms:*",
                "route53:*",
                "lambda:*",
                "ec2:*",
                "cognito-idp:*",
                "ebs:*",
                "eks:*",
                "acm:*",
                "rds:*"
            \],
            "Resource": "*"
        \}
    \]
\}

1. Check out repository {+}https://bitbucket.org/senofi/openidl-devops/src/develop/+^{Image Removed}
2. Create a new folder under openidl-devops/aws-infrastructure/environments/ by copying the sample folder openidl-devops/aws-infrastructure/environments/openidl-sample
3. Installed Terraform
4. Installed terragrunt
5. Terraform Cloud account. Register or sign in at {+}https://app.terraform.io+^{Image Removed}
6. Preconfigured access in ~/.terraformrc . Get the token from {+}https://app.terraform.io+^{Image Removed} by going to Settings → Teams → Team API Token. Generate a new token and create the file ~/.terraformrc

credentials "app.terraform.io" {
token = "iz5o8MNxgBBPwQ...."
}
Make sure there are no other credentials in the ~/.terraform/ folder as they will override the ones in file ~/.terraformrc

Procedure

The terraform/terragrunt scripts in openid-devops Git repository are used to create the required IAM roles and users, and the Kubernetes cluster that will host Jenkins and Ansible Tower. The modules' backend is configured to use Terraform Cloud.
Terragrunt executes the scripts via the preconfigured backend. The backend is configured to run Terraform from Terraform Cloud (Execution mode: remote) so the AWS credentials have to be set under the workspace's variables as Environment variables:
Image Removed
The backend (Terraform Cloud workspace) is created and initialized by Terragrunt automatically

Create IAM Role and User

1. Pull the AWS credentials from aws.senofi.ca for the AWS account created. They could be set in the console by choosing the 'Command line or programmatic access' link. Paste in the command line the values:

export AWS_ACCESS_KEY_ID="ASIAWRO3..."
export AWS_SECRET_ACCESS_KEY="6gLd..."
export AWS_SESSION_TOKEN="IQoJpZ2l..."

1. Go to openidl-devops/aws-infrastructure/environments/<env-folder> as copied in prerequisites section
2. Configure openidl-devops/aws-infrastructure/environments/<env-folder>/org-vars.yaml
3. Go to <env-folder>/iam and run terragrunt plan
4. After a review apply the changes with terragrunt apply

The script creates:

• IAM role (for the terraform user)
• IAM user (terraform user)

Create K8s Cluster

1. Register manually a new key pair in AWS by going to EC2 → Key pairs. Create a new key with name awx-target Keep the private key in the environments folder
2. Go to the workspace that was just created in the previous section and go to States tab. Open the top state in the list and find outputs and copy access_key and secret_key values that will be used for the next step
3. Go to <env-folder>/k8s-cluster and run terragrunt plan
4. The previous step should fail but it should have created a new workspace in Terraform Cloud - e.g. devnet-d3-k8s-cluster
5. Make sure the AWS variables are set in Terraform Cloud as Terraform variables under the workspace in the previous step → Variables tab
   1. aws_access_key = terraform user's access key ID
   2. aws_secret_key = terraform user's secret access key
   3. region = us-east-2 or any other region you prefer
   4. aws_role_arn = terrform role ARN
   5. aws_external_id = terraform
6. Run again terragrunt plan
7. Review and if things look ok run terragrunt apply
8. Acknowledge the run with yes in the prompt

The script creates:

• Kubernetes cluster
• PostgreSQL DB for Ansible Tower
• VPC, network

Import the Kubernetes Cluster connection config

1. Make sure you have AWS profile or key/secret/session environment variables set onto the terminal session
2. Find the name of the Kubernetes cluster and update the local config with it

aws eks update-kubeconfig --name ops-k8s

Install Ingress

1. Install Nginx Ingress controller

kubectl create ns ingress-nginx
helm install -n ingress-nginx lb ingress-nginx/ingress-nginx
It is possible that the nginx LB will not be assigned DNS and IP due to the security group for the cluster and the nodes tagged with the same annotation. To fix that find the security group for the nodes (e.g. ops-k8s-node) and remove the owned tag.

Install Jenkins

Use the helm chart for installing Jenkins onto the Kubernetes cluster created above.
cd <devops-repo>/jenkins
kubectl create ns jenkins
helm repo add jenkins https://charts.jenkins.io^{Image Removed}
helm upgrade --install -n jenkins jenkins jenkins/jenkins --values values.yaml
Wait for Jenkins to start up. 
To view the Jenkins admin password:
kubectl exec --namespace jenkins -it svc/jenkins -c jenkins – /bin/cat /run/secrets/additional/chart-admin-password && echo
Set up a cloud provisioned Jenkins node as defined in the Kubernetes plugin config in Jenkins.

Install Ansible Tower (AWX)

Create the AWX DB by connecting to the RDS PostgreSQL instance created via Terraform.

1. Create an SSH Tunnel. Lookup the RDS DB DNS and the EC2 instance that is the AWX target public DNS and replace them in the command line template:

ssh -i <env-folder>/awx-target.pem -N -L 5432:ops-tools-db.<instance-id>.us-east-2.rds.amazonaws.com:5432 ubuntu@<awx-target-ec2>.us-east-2.compute.amazonaws.com -vv

1. Connect with DBeaver (or another PostgreSQL client) on localhost port 5432 and run the following SQL after replacing <pass> with an actual password (as defined under environments/<env>/org-vars.yaml )

create database awx;
create user awxuser with encrypted password '<pass>';
grant all privileges on database awx to awxuser;

1. Configure the kustomize script awx-custom.yaml by replacing the DB settings in awx-operator folder under openidl-devops Git repository.

Install AWX with the Kustomize script.
helm repo add awx-operator https://ansible.github.io/awx-operator/^{Image Removed}

1. clone the gitops repo and go to awx-operator kustomize
   cd awx-operator
   kustomize build . | kubectl apply -f -
   Watch for the script failing and if it does run it again (timing issue due to the creation of the AWX RBAC)

   Update DNS

2. Go to the AWS Account → Route53
3. Create a new Hosted Zone (e.g. d1.test.senofi.net)
4. Under the new hosted zone create a new entry of type A with an Alias for the Kubernetes cluster (e.g. ops.d1.test.senofi.net) to point to a Classic Load Balancer

Now Jenkins and AWX should be available via {+}http://ops.d1.test.senofi.net/+^{Image Removed} and {+}http://ops.d1.test.senofi.net/jenkins+^{Image Removed}.

Terraform Cloud Workspaces

We need to maintain two workspaces - one for the Fabric Kubernetes cluster and one for the openIDL applications.
To create the workspaces use the tool located in {+}https://bitbucket.org/senofi/openidl-devops/src/develop/+^{Image Removed}:

1. Go to openidl-devops/aws-infrastructure/environments/<env-folder>/terraform-cloud and run

terragrunt plan
If everything looks ok, execute terragrunt apply. This should create two workspaces and a var set in Terraform Cloud.

1. Create a new KMS key (symetric, encrypt/decrypt) in the AWS console. The name is not important but use a meaningful name that will associate it with this environment. Use it to populate the property in the next step
2. Go to openidl-devops/automation/terraform-cloud and update configuration.properties Make sure that the varset name
3. Create SSH keys

ssh-keygen -t rsa -f app_eks_worker_nodes_ssh_key.pem
ssh-keygen -t rsa -f blk_eks_worker_nodes_ssh_key.pem
ssh-keygen -t rsa -f bastion_ssh_key.pem

1. Populate the variable set by executing the following command in openidl-devops/automation/terraform-cloud

pip install -r requirements.txt
python populate-variable-set.py

1. Copy the contents of the public keys and populate them in Terraform Cloud UI under Variable Sets → <the newly created varset>

Configure Jenkins

1. Set Jenkins node label 'openidl' in Kubernetes Cloud by going to Manage Jenkins → Manage Nodes and Clouds → Configure Clouds. Make sure that under Pod Template details the labels field contains the value 'openidl'.

Image Removed
Also, remove the prepopulated 'sleep' command if it is set on the pod template:
Image Removed

1. Create the Terraform Job Template
   1. Terraform Token Secret - Login to Jenkins go to Manage Jenkins → Manage Credentials → Stores scoped to Jenkins (Jenkins) → Global Credentials (unrestricted) → Add credentials

Choose Kind as secret text, enter secret text like Token in "secret" field and name the secret ID as unique since it will be used in pipeline code.
Image Removed

1. 1. Git Credentials - Add a new credential
2. Terraform Job
   1. Go to Jenkins → New Item. Use a name such as Terraform Job
   2. Select job type as PIPELINE and proceed.
   3. Select Definition as Pipeline Script from SCM
   4. Select SCM as Git
   5. Key in the Infrastructure code repository (openidl-gitops) URL.
   6. Select the Git credential created above
   7. Specify the relevant branch "refs/heads/<branch-name>".
   8. Set script path to jenkins-jobs/jenkinsfile-tf

Run Terraform Job

1. Run the Jenkins Terraform Job
2. Open the console log for the job. Once the job asks for an input accept and choose the apply option
3. The job runs a second plan into the Kubernetes workspace in Terraform Cloud. When asked - accept and apply the changes
4. Go to the AWS Console and find EKS (Elastic Kubernetes Service). Choose the blk cluster and go to Add-Ons. Find the EBS plugin and add it to the list. The plugin makes sure volumes could be created in Kubernetes

Application Deployment

Application TLS Cert

1. Deploy cert manager using helm , include CRD deployment as part of the helm install command (Note 1.11.0 doesn't work on k8s 1.20) - {+}https://cert-manager.io/docs/installation/helm/+^{Image Removed}
2. Deploy an issuer (kubectl apply)

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: openidl@senofi.ca
privateKeySecretRef:
name: letsencrypt-prod
server: https://acme-v02.api.letsencrypt.org/directory^{Image Removed}
solvers:

...

1. Terraform Cloud Account

2. Preconfigured access in ~/.terraformrc . Get the token from https://app.terraform.io by going to Settings → Teams → Team API Token. Generate a new token and create the file ~/.terraformrc

   Code Block
   credentials "app.terraform.io" { token = "iz5o8MNxgBBPwQ...." }

   
   

   
   

aws eks update-kubeconfig --name ops-k8s

cd <devops-repo>/jenkins kubectl create ns jenkins helm repo add jenkins https://charts.jenkins.io helm upgrade --install -n jenkins jenkins jenkins/jenkins --values values.yaml

AWX Setup

Prerequisites:

• AWX is installed and operational

• AWS infrastructure is provisioned

• Configuration is done and available at a private git repository

• Credentials information is defined and available

  • aws iam user

  • git private repo deploy key

  • bastion ssh private key

  • hds db access

  • fabric console user and password

• Bastion machine (gateway) host address is available

Steps:

Credentials:

AWX Job Templates: