The API Server Console shows details of calls to the API Server workers, including configuration, load, trace of actions and logs.
The console is available on any iCM server directly via port 5706. As you probably don't have direct server access you can also access the console from within iCM, via the iCM/API Servers menu, and from the End Point Editor.
The console is proxied from your current iCM session and shows a header telling you which API Server you have connected to. You can open consoles for multiple servers at the same time.
Your iCM user will need the "Access API Server Console" privilege.
Overview
The overview page shows summary information including:
- The API Server version and build
- When the server was started
- For how long the server has been running
- When the workers were started - which may be different to the server start if the configuration has been re-sent
- Total worker requests since the workers started
- Total server requests since the workers started
- Processing requests
- Queued requests
Beneath the summary information is a panel showing the current status of the workers. This shows the running workers along with access counts and live sparklines graphs of the current requests. Should any errors have occurred on a worker, a total is shown highlighted in red.
Each worker also displays a colour-coded button which indicates load over the previous ten seconds. Loading is the number of concurrent requests a worker is handling. Green indicates a low load, yellow over 80%, amber 90% and red 100%.
Should load on a worker exceed 100% requests will begin queuing.
Worker Details
You can click on the name of a worker from the overview to display further details about that worker. This shows the name of the worker, followed by the total number of requests to it, and the number of errors since the worker last restarted.
The worker details section also displays the number of worker threads and instances, as set in the configuration file sent to the API Server. Processing and Queued requests, and Load percentage, show the current activity of the worker.
Underneath the title and count line is bar graph of the type and number of requests to the worker. You can sort the list by name or number of requests. A count of errors is also shown, if there have been any.
Clicking on the cog icon next to the worker name displays the configuration information for that worker.
Worker Configuration
While viewing the worker details, you can see the current configuration of a worker by clicking the cog icon. A worker's configuration is a combination of the default worker.conf and the configuration sent to the worker using the API Server Config file. See API Server and Worker Configuration for further information.
You can switch back to the worker details by clicking the bar graph icon.
The configuration is displayed in JSON format. You can collapse and expand blocks of JSON using the small + and - buttons.
Logs
The API Server generates several types of log file. You can download historic files and view current files in your browser.
Log File | Description |
---|---|
apiserver-date.log | A record of API server restarts and error messages logged by workers |
core-date.csv | A snapshot of API Server activity, logged every minute, in a csv format. The csv columns include counts of the total requests to the API Server, queued requests, open requests and errors |
http-date.log | A timestamped list of server and worker activity including originating IP addresses and user agent strings |
system-dump-date.json | These files are generated on demand using the "Create system dump file" button. They contain a complete snapshot of the API Server, including worker configuration, server configuration, statistics from the core and worker csv files, heap space statistics and memory usage. Searching the files for for "runningjobs" is often a good place to start. |
system-dump-shutdown-date.json | Contain the same information as the on demand system dumps, but generated whenever the API Server process stops or restarts gracefully |
system-dump-heavy-load.json | Generated when under heavy load, see the monitoring section below |
worker.csv | A snapshot of worker activity, including the same counts as the core files, but just for this worker |
Additional log files may also be present if you have End Points using the
Trace
The Trace display lets you to see the requests being made to workers. This can be either a snapshot of the last 10 or 30 minutes, or a running display of requests being made in near real time.
Trace messages can be filtered by level, source and/or a search string. The search string is a case insensitive word or continuous phrase. This filtering applies to any trace messages that have been received by the console - so changing the filter will show or hide any messages that have been captured.
The Trace display shows events in order, with the most recent at the top. Clicking on the cog icon below the trace message opens a pop up window showing the Trace Details.
Trace Settings
The trace settings section enables additional tracing for the selected workers, and can be tailored to only show messages recorded at a set level (ie just warnings, or just errors). This type of tracing is in addition to any enabled in the End Point and forms designers.
Check the boxes next to the workers you'd like to enable tracing for and set the level of messages you'd like to capture. Once this has been set, click the "Activate" button. Tracing remains active for fifteen minutes.
When enabled you'll be able to see requests to individual workers and the responses they return. This level of tracing can be useful for debugging or troubleshooting workers, forms, End Points, workflows and API Server client code.
Trace Settings for workers can be useful in situations where multiple workers are involved. For example, an End Point with tracing enabled, deployed to the Ajax Library, making a request to the Workflow Worker's getHistoricProcessInstances method, will only show the activity of the End Point in the standard tracing. Using Trace Settings to enable tracing for the Ajax Library and Workflow Worker would allow you to follow the request to the Ajax Library, the request to the Workflow Worker, the Workflow Worker's response and finally the Ajax Library response.
Monitoring and Long Running Requests
The API Server automatically logs warnings for long running requests and generates dump files when under heavy load. The standard settings are:
"monitor": {
"interval": 5000, // The rate the monitor should tick
"longRunningTime": 120000, // How long a request should run before we log a warning
"longRunningRequestsLoggedTTL": 300000, // How long to wait before relogging a request
"heavyLoadMonitorTicksBeforeLog": 3, // How many monitor ticks of high load should pass before we log
"heavyLoadDumpRate": 60000, // What's the minimum time we should wait between dump files
"heavyLoadSize": 300 // The load (%) to warn and perform a dump at
}