Server Work Management
Fabric3 processes all runtime tasks and requests via a single thread pool. This provides centralized work management and statistics for all runtime operations. The thread pool is a specialized implementation of java.util.concurrent. AbstractExecutorService. The following attributes can be configured on the <thread.pool> element in systemConfig.xml:
- coreSize – the number of threads to keep in the pool, even if they are idle.
- size - the maximum number of threads to allow in the pool.
- queueSize – the size of the queue used to receive work requests. If an attempt is made to schedule a task when the queue has reached is maximum size, the task will be rejected. The default is 100,000.
- keepAliveTime - when the number of threads is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminating.
- allowCoreThreadTimeOut - If false (default) core threads stay alive even when idle. If true, core threads use keepAliveTime to time out waiting for work.
- rejected.execution.handler - One of the following values corresponding to the appropriate JDK RejectedExecutionHandler: abort, discard, discard.oldest, caller.runs (default).
- statistics.off: – false by default. If set to true, the runtime will collect work statistics.
- checkStalledThreads – true if the runtime should check for stalled threads. When a stalled thread is encountered, the runtime will emit a warning event with a stack trace to assist in identifying the current executing code. If statistics are off, this value will be ignored.
- stallThreshold – the threshold in milliseconds after which a thread processing a request is considered stalled.
- stallCheckPeriod – the time in milliseconds between checking for stalled threads.
Note it is important to understand the relationship between coreSize, size, and queueSize. As explained in detail in the Javadoc for java.util.concurrent.ThreadPoolExecutor, a thread will not be added to the thread pool until the core size has been reached and the work queue has reached the queueSize capacity. This may lead to tasks not being performed as the number of core threads has been reached but the maximum queue size has not. In this case, the runtime will not add additional threads to process work. The correct approach is to set the coreSize to the maximum pool size and allowCoreThreadTimeOut to true. This will allow the thread pool to grow even if the maximum queue size has not been reached and non-utilized threads to be discarded.
By default starting in Fabric3 1.9.7, the thread pool is configured to use the ThreadPoolExecutor.CallerRunsPolicy when an overflow condition occurs. In most cases this is a sensible choice as rejected work will push back on the client. For requests originating from a remote client, this will result in push back at the transport layer (e.g. TCP). For local-origin requests, push back will be applied directly to the client.
The runtime thread pool can be managed using JMX or the The Management Resource Framework (MRF). The MRF address is http://<server address>:<port>/management/runtime/threadpool.