Load Generation System for Trade Matching Systems

Operation

Users select one of the following options from thew Test Management Portal:

Generate a new sequence of json files as load requests in a range 1 to n.
Load pre-existing messages from a kafka topic (sequence n to m) as inputs to a new load test
Restart producer / consumer kubernetes pods to trigger the load test run
Back up files from previous load test (to use in future load test if needed)
Load historical load-test data for input to new load test
Access Grafana metrics of previous load tests

Load test activity:

Upon restart, producer pods consume thye generated data files from "/datastore/" and submit to the "messages" kafka topic.
Consumer pods read JSON messages from the "messages" topic and submit them as HTTP JSON requests via REST to the target ("load sink")

Producers, consumers and management nodes are implemented as containerised golang services in kubernetes pods. Each service is instrumented with the following custom metrics for evaluating the load testing activity:

Messages generated for input
Messages read and produced to the topic by producers
Producer errors generated in submitting requests to "messages" topic
Consumer errors generated in submitting requests to target api
Rate of message submission to the target API (by consumers)

Troubleshooting procedure

View the requests received by the target application (The "black box")

make a manual request:

POST /orders HTTP/1.1 > Host: 10.42.0.199 > User-Agent: curl/7.74.0 > Accept: */* > Content-Type: application/json > Content-Length: 98 > * upload completely sent off: 98 out of 98 bytes * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Date: Sun, 26 Dec 2021 11:03:22 GMT < Content-Length: 0 < * Connection #0 to host 10.42.0.199 left intact">

[email protected]:/opt/kafka# curl -v   -X POST  http://10.42.0.199:80/orders   -H 'Content-Type: application/json'   -d '{"Name":"newOrder", "ID":"78912","Time":"223232113111","Data":"new order", "Eventname":"newOrder"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 10.42.0.199:80...
* Connected to 10.42.0.199 (10.42.0.199) port 80 (#0)
> POST /orders HTTP/1.1
> Host: 10.42.0.199
> User-Agent: curl/7.74.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 98
> 
* upload completely sent off: 98 out of 98 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sun, 26 Dec 2021 11:03:22 GMT
< Content-Length: 0
< 
* Connection #0 to host 10.42.0.199 left intact

check the error and request count by viewing the admin portal on the dummy app

Anvil Management Portal

Total Requests: 2 Total Errors: [email protected]:/opt/kafka# ">

[email protected]:/opt/kafka# curl --user "admin:Crypt0N0m1c0n##" http://10.42.0.199:80/admin
Anvil Management Portal
 Total Requests: 2 Total Errors: [email protected]:/opt/kafka#

(you may need to do this from inside a test container if connectivity outside the kubernetes cluster is not configured)

Development Tasks

[p] Scalable producer Pool
 [x] Create a Golang Service
  [?] Create a service for the duration of the run
  [x] Consume formatted input data in concurrent mode.
  [x] Publish to kafka Topic
  [x] move processed files to an output queue
  [] Ensure Effective Scaling
   [] Resolve concurrent access problems
    
   [] Scaling of worker processes
   [] job allocation scaling
    [] job allocation: distribute range of inputs among workers
     [] 1. scan the input folder
     [] 2. divide the batch into ranges among the workers
     [] 3. assign  the task  to the workers to loop over each batch

[p] Implement Scalable consumer pool
 [x] Create main worker service
 [x] Consume from messages queue from Kafka
 [x] Submit message via an API (PoC: write to output files in directory)
 
 [] Ensure Effective Scaling
  [] understand Golang Coroutines and channels properly
   [] seek a better code implementation
    [] The case for worker pools with goroutines: https://brandur.org/go-worker-pool

[] Scale Golang operations Concurrently 
 [] Test Scalability
  [] Configure loads
  [] Extract Metrics
  [] Analyse and Present Metrics

[] Baseline Configuration of System
[] Realtime Control of System

Kafka Configuration

messages
metrics
deadLetter

Image Build

https://docs.docker.com/language/golang/build-images/

Registry login (Assuming we're using ECR):

aws ecr get-login-password --region ap-southeast-1 | docker login --username AWS --password-stdin 605125156525.dkr.ecr.ap-southeast-1.amazonaws.com

Simply:

go mod init producer
docker build --tag thor:0.0.5 .

Or using the custom build script with registry config (login prior to this):

(base) [email protected] producer % ./build.sh -n load-tester -t 0.0.1 -p true -d true
Building docker image with options: 
Name: hammer
Tag: 0.0.1
Push: true
building image hammer with tag 0.0.1 and pushing to 605125156525.dkr.ecr.us-east-2.amazonaws.com repo hammer
running build command docker build --tag hammer:0.0.1 .
[+] Building 4.5s (19/19) FINISHED                                                                                                                                                                                                                                                                                 
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                          0.0s
 => => transferring dockerfile: 37B                                                                                                                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                               0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                                                                                                                                                    2.6s
 => CACHED docker-image://docker.io/docker/dockerfile:[email protected]:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                                                                                                                                                                               0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/golang:1.16-alpine                                                                                                                                                                                                                                         1.5s
 => [ 1/10] FROM docker.io/library/golang:[email protected]:41610aabe4ee677934b08685f7ffbeaa89166ed30df9da3f569d1e63789e1992                                                                                                                                                                                 0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                             0.0s
 => => transferring context: 84B                                                                                                                                                                                                                                                                              0.0s
 => CACHED [ 2/10] WORKDIR /app                                                                                                                                                                                                                                                                               0.0s
 => CACHED [ 3/10] COPY go.mod ./                                                                                                                                                                                                                                                                             0.0s
 => CACHED [ 4/10] COPY go.sum ./                                                                                                                                                                                                                                                                             0.0s
 => CACHED [ 5/10] RUN go mod download                                                                                                                                                                                                                                                                        0.0s
 => CACHED [ 6/10] COPY *.go ./                                                                                                                                                                                                                                                                               0.0s
 => CACHED [ 7/10] RUN go build -o /producer                                                                                                                                                                                                                                                                  0.0s
 => CACHED [ 8/10] RUN mkdir -p /datastore                                                                                                                                                                                                                                                                    0.0s
 => CACHED [ 9/10] RUN mkdir -p /processed                                                                                                                                                                                                                                                                    0.0s
 => CACHED [10/10] RUN mkdir -p /applogs/                                                                                                                                                                                                                                                                     0.0s
 => exporting to image                                                                                                                                                                                                                                                                                        0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                       0.0s
 => => writing image sha256:86883e6db410e787713bd09601461e7cd28aa3978c0ea01998d022c1940605d9                                                                                                                                                                                                                  0.0s
 => => naming to docker.io/library/hammer:0.0.1                                                                                                                                                                                                                                                               0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

pushing docker image: docker push 605125156525.dkr.ecr.us-east-2.amazonaws.com/hammer:0.0.1The push refers to repository [605125156525.dkr.ecr.us-east-2.amazonaws.com/hammer] 5d174fa0eb70: Preparing 549a79b81723: Preparing 16840b8df015: Preparing c4027ae51b81: Preparing 903fa0d61f6c: Preparing 88971a11698b: Preparing 00a4fb3dbb29: Preparing 35c63a883fd2: Preparing aa147b7c1d1f: Preparing 19c4d4cefc09: Preparing 46e96c819e17: Preparing b6f786c730a9: Preparing 63a6bdb95b08: Preparing 8d3ac3489996: Preparing 35c63a883fd2: Waiting aa147b7c1d1f: Waiting 88971a11698b: Waiting 00a4fb3dbb29: Waiting 19c4d4cefc09: Waiting 8d3ac3489996: Waiting b6f786c730a9: Waiting 46e96c819e17: Waiting 16840b8df015: Layer already exists 549a79b81723: Layer already exists c4027ae51b81: Layer already exists 903fa0d61f6c: Layer already exists 5d174fa0eb70: Layer already exists 88971a11698b: Layer already exists 00a4fb3dbb29: Layer already exists 35c63a883fd2: Layer already exists aa147b7c1d1f: Layer already exists 19c4d4cefc09: Layer already exists 46e96c819e17: Layer already exists b6f786c730a9: Layer already exists 8d3ac3489996: Layer already exists 63a6bdb95b08: Layer already exists 0.0.1: digest: sha256:957908eec42149b1a6d7aa9dff955e3a71d1abc20a9d3720f2734946f0e640a5 size: 3240

REPOSITORY                                            TAG       IMAGE ID       CREATED        SIZE
REDACTED.dkr.ecr.us-east-2.amazonaws.com/hammer   0.0.1     86883e6db410   3 hours ago    544MB

Deploy to kubnernetes:

(base) [email protected] producer % kubectl apply -f producer-manifest.yaml
persistentvolumeclaim/datastore-claim created
persistentvolumeclaim/processed-claim created
persistentvolumeclaim/applogs-claim created
deployment.apps/producer created

Access to the registry

For AWS ECR:

Get the login credential

aws ecr get-login-password --region us-east-2

Create the secret using the docker-password obtain from the last step:

kubectl create secret docker-registry ragnarok \
    --namespace ragnarok \
    --docker-server=REDACTED.dkr.ecr.us-east-2.amazonaws.com \
    --docker-username=AWS \
    --docker-password="REDACTED"

Technical Challenges

[] Scheme for concurrent reading of a directory of input files
[] Too many file handles open connecting to Kafka
[] Matching concurrency to System Resources
[] How to customise payload to the target api (Allow for API customisation for loading)

Troubleshooting

incorrect message format (not readable json)unexpected end of JSON input
Error Count:  81
wrote:   to topic  metrics
panic: worker4could not create message payload for API: open ../output-api/37a80a7d-cb04-4a44-b3f7-5e39d40d93634: too many open files

goroutine 9 [running]:
main.consume_payload_data(0x13bd860, 0xc00001a0a8, 0x135be59, 0x8, 0x4, 0x0, 0x0)
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/consumer.go:163 +0xa3f
main.worker(0x4, 0xc0001d0000, 0xc0001e4000)
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/consumer.go:188 +0x85
created by main.main
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/consumer.go:213 +0x9f

error  open ../datastore/123: no such file or directory skipping over  123
panic: could not write message dial tcp 127.0.0.1:9092: socket: too many open filesto topicmessages

goroutine 33 [running]:
main.produce(0x0, 0x0, 0x13440c0, 0xc0000a2000, 0x12ecec9, 0x8)
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/producer.go:96 +0x465
main.process_input_data(0xf, 0x79, 0xc00062ff01, 0x2)
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/producer.go:61 +0x2c5
main.worker(0xf, 0xc000184000, 0xc000188000)
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/producer.go:118 +0x4f
created by main.main
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/producer.go:143 +0x9f

(base) [email protected] microservices % ./producer          
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1266e3a]

goroutine 41 [running]:
main.process_input_data(0x3, 0x4, 0x1)
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/producer.go:91 +0x37a
main.worker(0x3, 0xc0002a4000, 0xc0002a40b0)
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/producer.go:166 +0x5d
created by main.main
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/producer.go:254 +0xff
(base) [email protected] microservices % ./producer

fatal error: concurrent map writes
fatal error: concurrent map writes

goroutine 14 [running]:
        goroutine running on other thread; stack unavailable
created by main.main
        /Users/welcome/Desktop/STUFF/umbrellacorp/lolcorp-load-generator/microservices/producer.go:288 +0xff

fatal error: concurrent map writes

Technical Notes

Internal DNS Endpoints

By convention kubernetes will assign the following fqdns to the services in the relevant namespaces:

1. Kafka broker URL                    : kafka.kafka.svc.cluster.local:9092
2. Producer Service (metrics port)     : producer-service.ragnarok.svc.cluster.local:80
3. Consumer Service (metrics port)     : consumer-service.ragnarok.svc.cluster.local:80
4. Sink Service (API and metrics port) : sink-service.ragnarok.svc.cluster.local:80
5. NFS Service                         : nfs-service.ragnarok.svc.cluster.local: 2049

Installing kafka on k8s:

(base) [email protected] kafka % helm install kafka incubator/kafka -f values.yaml --namespace kafka
WARNING: This chart is deprecated
W1222 13:48:56.124818    2637 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1222 13:48:56.651731    2637 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
NAME: kafka
LAST DEPLOYED: Wed Dec 22 13:48:55 2021
NAMESPACE: kafka
STATUS: deployed
REVISION: 1
NOTES:
### Connecting to Kafka from inside Kubernetes

You can connect to Kafka by running a simple pod in the K8s cluster like this with a configuration like this:

  apiVersion: v1
  kind: Pod
  metadata:
    name: testclient
    namespace: kafka
  spec:
    containers:
    - name: kafka
      image: confluentinc/cp-kafka:5.0.1
      command:
        - sh
        - -c
        - "exec tail -f /dev/null"

Once you have the testclient pod above running, you can list all kafka
topics with:

  kubectl -n kafka exec testclient -- ./bin/kafka-topics.sh --zookeeper kafka-zookeeper:2181 --list

To create a new topic:

  kubectl -n kafka exec testclient -- ./bin/kafka-topics.sh --zookeeper kafka-zookeeper:2181 --topic test1 --create --partitions 1 --replication-factor 1

To listen for messages on a topic:

  kubectl -n kafka exec -ti testclient -- ./bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test1 --from-beginning

To stop the listener session above press: Ctrl+C

To start an interactive message producer session:
  kubectl -n kafka exec -ti testclient -- ./bin/kafka-console-producer.sh --broker-list kafka-headless:9092 --topic test1

To create a message in the above session, simply type the message and press "enter"
To end the producer session try: Ctrl+C

If you specify "zookeeper.connect" in configurationOverrides, please replace "kafka-zookeeper:2181" with the value of "zookeeper.connect", or you will get error.
(base) [email protected] kafka %