Setting up a JFrog Artifactory 7 and Xray 3 Sandbox in AWS Using minikube and Helm Charts

The JFrog Artifactory and its complementary suite of tools is well known across the industry. As part of a certification preparation, I wanted to find out more about how it is administered. This post is how to install JFrog Artifactory 7 and Xray 3 using Helm Charts in an AWS EC2 instance.

Prerequisites

You should know how to install the following for your OS:

  • Docker – the container runtime
  • Kubernetes – the container-orchestration system
  • Minikube – used to set up a single-node Kubernetes cluster on your local machine
  • Helm – a package manager for Kubernetes

You would also need your existing Artifactory and Xray license keys – or get an evaluation key.

Setup

As the intention is to install the sandbox on our local machine, I found that the smallest AWS EC2 instance size would be 4 vCPUs and 16 GB of RAM. For this purpose, I used a m5a.xlarge instance running an Ubuntu Server 20.04 LTS AMI.

There are two ways to install Artifactory 7 and Xray 3:

  • Install applications as separate Helm Charts (artifactory-oss and xray) and connect them using the join key
  • Install both applications under a single Helm Chart (jfrog-platform) that does the ordering and integration together. Do note that this Helm chart is in beta at time of writing this post.

Refer to the following paragraph from the JFrog documentation on when to use each approach:

When would you use the JFrog Platform installation versus installing JFrog products individually using the installations below? The most common use case would be if you want to install several product (or all of them) at the same time; using this installation saves you the need to repeat steps, repeating configuration of join keys and more, making the installation process easy and straightforward.

https://www.jfrog.com/confluence/display/JFROG/Installing+the+JFrog+Platform+Using+Helm+Chart

The latest Helm Chart versions available today are:

  • jfrog/artifactory-oss – 3.3.1
  • jfrog/xray – 6.6.0
  • jfrog/jfrog-platform – 0.0.12 (currently in beta)

Step 1: Install Docker, Kubernetes, Minikube and Helm

You should install these tools for your OS. Refer to pre-requisite section for links to instructions.

Before starting minikube, increase the memory limits to at least 8 GB. As I was using an instance with 16 GB of memory, I set aside 25% for the host OS (4 GB) and dedicated the rest to minikube:

$ minikube config set memory 12288

(Re)start minikube after increasing the memory limits.

$ minikube delete  ## only if you have running minikube instance
$ minikube start
minikube v1.15.1 on Ubuntu 20.04
Automatically selected the docker driver
Starting control plane node minikube in cluster minikube
Pulling base image …
Downloading Kubernetes v1.19.4 preload …
> preloaded-images-k8s-v6-v1.19.4-docker-overlay2-amd64.tar.lz4: 486.35 MiB
Creating docker container (CPUs=2, Memory=12288MB) …
Preparing Kubernetes v1.19.4 on Docker 19.03.13 …
Verifying Kubernetes components…
Enabled addons: storage-provisioner, default-storageclass
Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Step 2: Add JFrog Helm Repo

We want to be able to get the necessary Helm Charts from the JFrog Helm repo, so we would add it in and update the package manager:

$ helm repo add center https://repo.chartcenter.io
"center" has been added to your repositories

$ helm repo update 
Hang tight while we grab the latest from your chart repositories……
Successfully got an update from the "center" chart repository
Update Complete. ⎈Happy Helming!⎈

Step 3: Install Artifactory and Xray

Method A: Install Artifactory and Xray as Separate Helm Charts

Step A-1: Install Artifactory 7 Helm Chart

Create a namespace for the Artifactory 7 application using kubectl, and install the artifactory-oss chart using it.

For simplicity, I chose to:

  • supply custom variable values via the “set” flag inline,
  • use the embedded derby database (by setting postgresql.enabled=false), and
  • chose NodePort instead of LoadBalancer for the nginx service as stated in its values.yaml file
$ kubectl create namespace artifactory-oss
namespace/artifactory-oss created

$ helm install artifactory-oss center/jfrog/artifactory \
   --namespace artifactory-oss \
   --set postgresql.enabled=false \
   --set nginx.service.type=NodePort

NAME: artifactory-oss
 LAST DEPLOYED: Thu Dec 10 02:11:47 2020
 NAMESPACE: artifactory-oss
 STATUS: deployed
 REVISION: 1
 TEST SUITE: None
 NOTES:
 Congratulations. You have just deployed JFrog Artifactory!

1. Get the Artifactory URL by running these commands:
 export NODE_PORT=$(kubectl get --namespace artifactory-oss -o jsonpath="{.spec.ports[0].nodePort}" services artifactory-oss-artifactory-nginx)
 export NODE_IP=$(kubectl get nodes --namespace artifactory-oss -o jsonpath="{.items[0].status.addresses[0].address}")
 echo http://$NODE_IP:$NODE_PORT/

2. Open Artifactory in your browser
 Default credential for Artifactory:
 user: admin
 password: password 

Validate that all pods have been created successfully:

$ kubectl get pods -n artifactory-oss
NAME                                                 READY   STATUS    RESTARTS   AGE
artifactory-oss-artifactory-0                        1/1     Running   0          2m8s
artifactory-oss-artifactory-nginx-7f77999d9b-kzhz4   1/1     Running   0          2m15s

Step A-2: Add Security Group Inbound Rules

Get the port mapping used by the nginx NodePort:

$ kubectl get services -n artifactory-oss
NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
artifactory-oss-artifactory         ClusterIP   10.106.177.248           8082/TCP,8081/TCP            26m
artifactory-oss-artifactory-nginx   NodePort    10.97.75.160             80:31877/TCP,443:32487/TCP   26m

Based on the output above, we can see that all traffic on port 80 and 443 is being routed to nginx pod 31877 and 32487 respectively.

Add the latter ports to the AWS EC2 instance’s security group inbound rules. In my example, I added 31877 and 32487 to the security group inbound rules to allow from “anywhere”, you should do the same but with your port numbers.

Step A-3: Setup Port Forwarding

As we want to access the sandbox from the Internet, we need to forward all traffic from the NodePort port to the nginx pod. For now, we will forward traffic to the HTTP port:

$ kubectl port-forward service/artifactory-oss-artifactory-nginx 31877:80  -n artifactory-oss --address 0.0.0.0
Forwarding from 0.0.0.0:31877 -> 80

Do note that the kubectl port-forward “is not particularly reliable for long-running connections” – see this github issue for more details.

Step A-4: Access Artifactory via EC2 Instance Public IP

If all goes well, you should be able to see the Artifactory 7 landing page by typing in EC2 instance’s public IP address and port X used by the kubectl port-forward in Step A-3. Use the default credentials in Step A-1 to login, and as part of the sign in process:

Step A-5: Retrieve Artifactory Join Key and Base URL

We need the join key and base URL to set up Xray 3 – the join key is a trusted token from Artifactory 7 that allows Xray to connect with it. You can find from the “Administration > Security > Settings > Connection details” menu from the left.

This method is different than the previous major release of Artifactory 6 and Xray 2, which required going to a separate Xray GUI and entering the Artifactory URL and admin user credentials.

Step A-6: Install Xray 3 Helm Chart

Note: you might need a separate terminal for this while kubectl port-forward is running, or send it to background.

Now that the Artifactory application is running, we can turn our focus to Xray. Just like last time, create a namespace for the Xray application and install its Helm Chart, passing in the joinKey and base URL from the previous step:

$ kubectl create namespace xray
namespace/xray created

$ helm install xray center/jfrog/xray \
   --namespace xray \
   --set postgresql.postgresqlPassword=password \
   --set xray.joinKey=<YOUR_ARTIFACTORY_JOIN_KEY> \
   --set xray.jfrogUrl=<YOUR_ARTIFACTORY_BASE_URL>

NAME: xray
LAST DEPLOYED: Thu Dec 10 03:04:37 2020
NAMESPACE: xray
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Congratulations! JFrog Xray services are deployed!
***** WARNING *****
Your Xray master key is still set to the provided example:
xray.masterKey=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 
    
You should change this to your own generated key:                                    $ export MASTER_KEY=$(openssl rand -hex 32)
$ echo ${MASTER_KEY}

Pass the created master key to helm with '--set xray.masterKey=${MASTER_KEY}'       **** 
 
Open Artifactory - <YOUR_ARTIFACTORY_BASE_URL> in your browser. Xray should be activated with Artifactory.

Note the master key warning above, but can be ignored for this sandbox.

Check that the Xray 3 pods have been created successfully:

$ kubectl get pods -n xray
NAME                 READY   STATUS    RESTARTS   AGE
xray-0               5/5     Running   0          73s
xray-postgresql-0    1/1     Running   0          114s
xray-rabbitmq-ha-0   1/1     Running   0          2m25s

Step A-7: Enter Xray 3 License Key

Back in the administration page of Artifactory 7, navigate to “Administration > Licenses > Activate Xray Trial License” and enter your license key in the textbox provided. If this option does not appear, you might need to refresh your page.

Once the license has been accepted, refresh the page and you should be able to access the “Application > Security & Compliance” menu on the left of the page.

Step A-8: Uninstall (Optional)

If you need to remove the release(s), run the following command(s) to do so:

$ helm uninstall xray --namespace xray 
$ helm uninstall artifactory-oss --namespace artifactory-oss

Method B: Install Artifactory and Xray Using the Jfrog Platform Helm Chart

Step B-1: Create a Custom Value YAML File

The JFrog Platform Helm Chart uses many of the individual product charts (as sub-charts) with almost all of them enabled at the start. It will be easier to understand if we wrote a separate custom value YAML and passed it to Helm.

I named my file custom-values.yaml and this is the content:

$ cat custom-values.yaml
global:
  jfrogUrl: "http://{{ .Release.Name }}-artifactory:8082"
  jfrogUrlUI: "http://{{ .Release.Name }}-artifactory:8082"
redis:
  enabled: false
distribution:
  enabled: false
mission-control:
  enabled: false
pipelines:
  enabled: false
artifactory-ha:
  enabled: false
artifactory:
  enabled: true
  nginx:
    service:
      type: NodePort
  license:
    licenseKey: |-
      <ARTIFACTORY_LICENSE_KEY>

Note: if you are running artifactory-ha instead, do remove the custom jfrogUrl and jfrogUrlUI keys in global block.

Step B-2: Install Artifactory 7 and Xray 3 Using the Jfrog Platform Helm Chart

Using a single command (one of the advantages of using the jfrog-platform Helm Chart), install both applications:

$ kubectl create namespace jfrog-platform
namespace/jfrog-platform created

$ helm upgrade --install jfrog-platform --namespace jfrog-platform center/jfrog/jfrog-platform -f custom-values.yaml

Release "jfrog-platform" does not exist. Installing it now.
NAME: jfrog-platform
LAST DEPLOYED: Thu Dec 10 03:26:56 2020
NAMESPACE: jfrog-platform
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Congratulations. You have just deployed JFrog Platform Chart with following products:
- artifactory
- xray 

1. Get the Artifactory URL by running these commands:
 export NODE_PORT=$(kubectl get --namespace jfrog-platform -o jsonpath="{.spec.ports[0].nodePort}" services jfrog-platform-artifactory-nginx)
 export NODE_IP=$(kubectl get nodes --namespace jfrog-platform -o jsonpath="{.items[0].status.addresses[0].address}")
 echo http://$NODE_IP:$NODE_PORT/

2. Open Artifactory in your browser
 Default credential for Artifactory:
 user: admin
 password: password 

Using the buldled postgresql database:
To extract the database password, run the following
export DB_PASSWORD=$(kubectl get --namespace jfrog-platform $(kubectl get secret --namespace jfrog-platform -o name | grep postgresql) -o jsonpath="{.data.postgresql-password}" | base64 --decode)
echo ${DB_PASSWORD}

Open Artifactory URL in your browser. Enabled services should be activated with Artifactory.

Check that all pods have been created successfully:

$ kubectl get pods -n jfrog-platform 
NAME                                                READY   STATUS    RESTARTS   AGE
jfrog-platform-artifactory-0                        1/1     Running   0          7m36s
jfrog-platform-artifactory-nginx-6499db75cf-phdql   1/1     Running   0          7m37s
jfrog-platform-postgresql-0                         1/1     Running   0          7m16s
jfrog-platform-rabbitmq-0                           1/1     Running   0          7m36s
jfrog-platform-xray-0                               5/5     Running   0          7m16s

Step B-3: Add Security Group Inbound Rules

Get the port mapping used by the nginx NodePort:

$ kubectl get services -n jfrog-platform | grep nginx
jfrog-platform-artifactory-nginx     NodePort    10.98.63.121             80:31629/TCP,443:31699/TCP              12m

Based on the output above, we can see that all traffic on port 80 and 443 is being routed to nginx pod 31629 and 31699 respectively.

Add the latter ports to the AWS EC2 instance’s security group inbound rules. In my example, I added 31629 and 31699 to the security group inbound rules to allow from “anywhere”, you should do the same but with your port numbers.

Step B-4: Setup Port Forwarding

As we want to access the sandbox from the Internet, we need to forward all traffic from the NodePort port to the nginx pod. For now, we will forward traffic to the HTTP port:

$ kubectl port-forward service/jfrog-platform-artifactory-nginx 31629:80  -n jfrog-platform --address 0.0.0.0
Forwarding from 0.0.0.0:31629 -> 80

Do note that the kubectl port-forward “is not particularly reliable for long-running connections” – see this github issue for more details.

Step B-5: Access Artifactory 7 via EC2 Instance Public IP

If all goes well, you should be able to see the Artifactory 7 landing page by typing in EC2 instance’s public IP address and port X used by the kubectl port-forward in Step A-3. Use the default credentials in Step A-1 to login.

If you have, like me, added the license key in the custom-values.yaml file, you should be shown the application dashboard immediately. Otherwise, you will be prompted to enter the Artifactory license key.

Step B-6: Enter Xray 3 License Key

Xray 3 is not activated by default. To do so, go to the administration page of Artifactory 7 and navigate to “Administration > Licenses > Activate Xray Trial License” and enter your license key in the textbox provided. If this option does not appear, you might need to refresh your page.

Once the license has been accepted, refresh the page and you should be able to access the “Application > Security & Compliance” menu on the left of the page.

Step B-7: Uninstall (Optional)

If you need to remove the release, run the following command to do so:

$ helm uninstall jfrog-platform --namespace jfrog-platform

Summary

That’s it – we managed to set up an Artifactory and Xray sandbox in minikube using the official Helm Charts in AWS, and have made it accessible from the public Internet.

Bonus – Exploring the Shared Database

When using the jfrog-platform chart to set up this sandbox of ours (i.e. Method B above), a single PostgreSQL database will be set up and shared between Artifactory and Xray. This is sufficient for non-production, but in the real world you would want to point these endpoints to an external one (e.g. a RDS PostgreSQL instance).

Let’s connect to the database to illustrate:

$ kubectl get pods -n jfrog-platform | grep postgresql
jfrog-platform-postgresql-0                         1/1     Running   0          25m

kubectl exec -n jfrog-platform --stdin --tty  jfrog-platform-postgresql-0  -- /bin/bash

I have no name!@jfrog-platform-postgresql-0:/$ psql -h localhost -U postgres
Password for user postgres:
psql (12.3)
Type "help" for help.

postgres=# SELECT datname FROM pg_database;
  datname
-----------
postgres
template1
template0
artifactory
xray
(5 rows)

postgres=#

Bonus – Helpful Troubleshooting Commands

Here are some of the commands that I used to troubleshoot my installation while writing this post:

Get a Shell to the container in a pod:

$ kubectl exec -n <NAMESPACE> --stdin --tty <CONTAINER_NAME>  -- /bin/bash

See the list of containers in the Xray pod:

$ kubectl describe -n <NAMESPACE> pod/xray-0 

Download a Chart and see what is in its values.yaml:

$ helm pull --untar center/jfrog/artifactory-oss

See logs of artifactory service:

$ kubectl exec -n <NAMESPACE> --stdin --tty <CONTAINER_NAME>  -- /bin/bash
artifactory-container$ tail -f var/log/artifactory-service.log

Artifactory cannot find cluster join key despite it being in /var/opt/jfrog/artifactory/bootstrap/access/etc/security/join.key?

$ kubectl exec -n jfrog-platform --stdin --tty jfrog-platform-artifactory-0  -- /bin/bash

artifactory$ tail -f var/log/artifactory-service.log
2020-12-10T03:28:35.867Z [jfrt ] [INFO ] [840cae9b048f609e] [o.j.s.c.KeyUtils:108          ] [art-init            ] - Cluster join: Join key is missing. Pending for 5 seconds with 60 seconds timeout

artifactory$ find / -name join.key 2>/dev/null
/var/opt/jfrog/artifactory/bootstrap/access/etc/security/join.key

artifactory$ cat /var/opt/jfrog/artifactory/bootstrap/access/etc/security/join.key
<VALID_JOIN_KEY>

Solution: Delete and start a new minikube local cluster.

$ minikube delete && minikube start

Need more time for pods to start up? Modify liveness and readiness checks:

$ helm install ... \
  --set *.livenessProbe.failureThreshold=999 \
  --set *.readinessProbe.failureThreshold=999 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s