The API Server Console provides a lot of information about worker and API Server activity. This article explains what it all means.
API Server Overview
The overview of API Server activity displays information about the API Server and its workers since it last started. Restarting the service will reset all counters and graphs. Logs are maintained as they are written to disk.
The traffic light buttons next to each worker indicate the load of each worker over the past ten seconds. Loading is the number of concurrent requests a worker is handling. Green indicates a low load, yellow over 80%, amber 90% and red 100% or more. One request will use one thread of one instance, so should a worker only have one instance and a single thread, one request would put the load at 100%, with further requests queued until the current request is processed.
A load over 100% is not in itself a concern. A worker under heavy load will automatically queue requests, meaning that what initially seems a heavy load can clear within a few seconds. Should there be sustained heavy load, with no recovery, it may be necessary to increase the number of instances, or threads, for that worker.
The sparkline graph shows the activity of a worker over the previous thirty seconds. The screenshot below shows a large number of requests to the Form Utilities Worker, and the sparkline graph how long ago the activity started. The Form Utilities Worker will be explored further below.
Worker Details
From the overview each worker can be examined in closer detail.
The information available from the worker detail view is sampled every two seconds. Queued requests and load percentage show the activity of that worker at the moment the sample was taken. The max values record the maximum load and queue since the worker last restarted.
The load bar graph samples every two seconds, scrolling to the left as new samples are taken. When examining the graph it is important not to become too focused on the peaks of activity. In the example above you can see several occasions where activity exceeded 100% and 200%, followed by recovery where all requests had been processed or there was no current activity.
At the far right of the graph is a period of sustained high load, peaking at 480%. Note that when 400% is exceeded the graph does not scale to fit the peak, but highlights the top of the bar in pink.
When considering the reasons for heavy load the API Server trace view should be examined. This will highlight any particularly long running requests that may be contributing the load. The trace includes the time that the request was made and the time of the response: