Onnx kill switch (#779)

Introduce kill switch mechanism for terminating long onnxruntime sessions
* Refactor background workers
* Refactor backends loading
* Add load time config for MODEL_EXECUTION_TIMEOUT. Parallel tests seems not to work.
* Refactor load time config
* More PR fixes, add the option to get the global run sessions array from backend len (supported only for onnx now) in INFO MODULES command.
* Return error if onnx is executed in a non async manner (via gears for instance).
test info command with AI fields

Author: alonre24

docs/commands.md

@@ -934,14 +934,14 @@ Because `AI.DAGRUN` provides the `PERSIST` option it is flagged as a 'write' com
     Refer to the Redis [`READONLY` command](https://redis.io/commands/readonly) for further information about read-only cluster replicas.
 
 ## AI.INFO
-The **`AI.INFO`** command returns general module information or information about the execution a model or a script.
+The **`AI.INFO`** command returns information about the execution of a model or a script.
 
-Runtime information is collected each time that [`AI.MODELRUN`](#aimodelrun) or [`AI.SCRIPTRUN`](#aiscriptrun) is called. The information is stored locally by the executing RedisAI engine, so when deployed in a cluster each shard stores its own runtime information.
+Runtime information is collected each time that [`AI.MODELEXECUTE`](#aimodelrun) or [`AI.SCRIPTEXECUTE`](#aiscriptrun) is called. The information is stored locally by the executing RedisAI engine, so when deployed in a cluster each shard stores its own runtime information.
 
 **Redis API**
 
 ```
-AI.INFO [<key>] [RESETSTAT]
+AI.INFO <key> [RESETSTAT]
 ```
 
 _Arguments_
@@ -951,15 +951,7 @@ _Arguments_
 
 _Return_
 
-For a module genernal information: An array with alternating entries that represent the following key-value pairs:
-
-* **Version**: a string showing the current module version.
-* **Low level API Version**:  a string showing the current module's low level api version.
-* **RDB Encoding version**: a string showing the current module's RDB encoding version.
-* **TensorFlow version**: a string showing the current loaded TesnorFlow backend version.
-* **ONNX version**: a string showing the current loaded ONNX Runtime backend version.
-
-For model or script runtime information: An array with alternating entries that represent the following key-value pairs:
+An array with alternating entries that represent the following key-value pairs:
 
 * **KEY**: a String of the name of the key storing the model or script value
 * **TYPE**: a String of the type of value (i.e. 'MODEL' or 'SCRIPT')

src/CMakeLists.txt

@@ -39,6 +39,7 @@ ADD_LIBRARY(redisai_obj OBJECT
         execution/parsing/parse_utils.c
         execution/run_info.c
         execution/background_workers.c
+        execution/run_queue_info.c
         execution/utils.c
         config/config.c
         execution/DAG/dag.c
@@ -88,6 +89,7 @@ ENDIF()
 IF(BUILD_ORT)
     ADD_LIBRARY(redisai_onnxruntime_obj OBJECT
             backends/onnxruntime.c
+            backends/onnx_timeout.c
             ${BACKEND_COMMON_SRC}
 )
 ENDIF()

src/backends/backedns_api.h

@@ -0,0 +1,30 @@
+#pragma once
+
+#include <stdint.h>
+
+/**
+ * @return The internal id of RedisAI current working thread.
+ * id range is {0, ..., <threads_count>-1}. If this is called from a non
+ * RedisAI BG thread, return -1.
+ */
+long (*RedisAI_GetThreadId)(void);
+
+/**
+ * @return The number of working threads in RedisAI. This number should be
+ * equal to the number of threads per queue (load time config) * number of devices
+ * registered in RedisAI (a new device is registered if a model is set to run on
+ * this device in AI.MODELSTORE command.
+ */
+uintptr_t (*RedisAI_GetThreadsCount)(void);
+
+/**
+ * @return The number of working threads per device queue (load time config).
+ */
+long long (*RedisAI_GetNumThreadsPerQueue)(void);
+
+/**
+ * @return The maximal number of milliseconds that a model run session should run
+ * before it is terminated forcefully (load time config).
+ * Currently supported only fo onnxruntime backend.
+ */
+long long (*RedisAI_GetModelExecutionTimeout)(void);