feature: implement the new ngx.run_worker_thread API to run Lua function in a seperated worker thread.

Author: kingluo

README.markdown

@@ -1162,6 +1162,7 @@ Directives
 * [lua_max_pending_timers](#lua_max_pending_timers)
 * [lua_max_running_timers](#lua_max_running_timers)
 * [lua_sa_restart](#lua_sa_restart)
+* [lua_worker_thread_vm_pool_size](#lua_worker_thread_vm_pool_size)
 
 
 The basic building blocks of scripting Nginx with Lua are directives. Directives are used to specify when the user Lua code is run and
@@ -3328,6 +3329,25 @@ This directive was first introduced in the `v0.10.14` release.
 
 [Back to TOC](#directives)
 
+lua_worker_thread_vm_pool_size
+------------------------------
+
+**syntax:** *lua_worker_thread_vm_pool_size <size>*
+
+**default:** *lua_worker_thread_vm_pool_size 100*
+
+**context:** *http*
+
+Specifies the size limit of the Lua VM pool (default 100) that will be used in the [ngx.run_worker_thread](#ngxrun_worker_thread) API.
+
+Also, it is not allowed to create Lua VMs that exceeds the pool size limit.
+
+The Lua VM in the VM pool is used to execute Lua code in seperate thread.
+
+The pool is global at Nginx worker level. And it is used to reuse Lua VMs between requests.
+
+[Back to TOC](#directives)
+
 Nginx API for Lua
 =================
 
@@ -3480,6 +3500,7 @@ Nginx API for Lua
 * [coroutine.wrap](#coroutinewrap)
 * [coroutine.running](#coroutinerunning)
 * [coroutine.status](#coroutinestatus)
+* [ngx.run_worker_thread](#ngxrun_worker_thread)
 
 
 [Back to TOC](#table-of-contents)
@@ -8960,6 +8981,134 @@ This API was first enabled in the `v0.6.0` release.
 
 [Back to TOC](#nginx-api-for-lua)
 
+ngx.run_worker_thread
+---------------------
+
+**syntax:** *ok, res1, res2, ... = ngx.run_worker_thread(threadpool, module_name, func_name, arg1, arg2, ...)*
+
+**context:** *rewrite_by_lua*, access_by_lua*, content_by_lua**
+
+**This API is still experimental and may change in the future without notice.**
+
+**This API is available only for Linux.**
+
+Wrap the [nginx worker thread](http://nginx.org/en/docs/dev/development_guide.html#threads) to execute lua function. The caller coroutine would yield until the function returns.
+
+Note that no ngx_lua API can be used in the `function_name` function of the `module` module since it is invoked in a separate thread.
+
+The first argument `threadpool` specifies the Nginx thread pool name defined by [thread_pool](https://nginx.org/en/docs/ngx_core_module.html#thread_pool).
+
+The second argument `module_name` specifies the lua module name to execute in the worker thread, which would return a lua table. The module must be inside the package path, e.g.
+
+```nginx
+
+ lua_package_path '/opt/openresty/?.lua;;';
+```
+
+The third argument `func_name` specifies the function field in the module table as the second argument.
+
+The type of `arg`s must be one of type below:
+
+* boolean
+* number
+* string
+* nil
+* table (the table may be recursive, and contains members of types above.)
+
+The `ok` is in boolean type, which indicate the C land error (failed to get thread from thread pool, pcall the module function failed, .etc). If `ok` is `false`, the `res1` is the error string.
+
+The return values (res1, ...) are returned by invocation of the module function. Normally, the `res1` should be in boolean type, so that the caller could inspect the error.
+
+This API is useful when you need to execute the below types of tasks:
+
+* CPU bound task, e.g. do md5 calculation
+* File I/O task
+* Call `os.execute()` or blocking C API via `ffi`
+* Call external Lua library not based on cosocket or nginx
+
+Example1: do md5 calculation.
+
+```nginx
+
+ location /calc_md5 {
+     default_type 'text/plain';
+
+     content_by_lua_block {
+         local ok, ret, md5_or_err = ngx.run_worker_thread("testpool", "calc_md5", "md5", ngx.var.arg_str)
+         if not ok then
+             ngx.say(ret)
+             return
+         end
+         if not ret then
+             ngx.say(md5_or_err)
+             return
+         end
+         ngx.say(md5_or_err)
+     }
+ }
+```
+
+`calc_md5.lua`
+
+```lua
+
+ local resty_md5 = require "resty.md5"
+ local resty_str = require "resty.string"
+
+ local function md5(str)
+     local md5 = resty_md5:new()
+     if not md5 then
+         return false, "md5 new error"
+     end
+
+     local ok = md5:update(str)
+     if not ok then
+         return false, "md5 update error"
+     end
+
+     local digest = md5:final()
+     return true, resty_str.to_hex(digest)
+ end
+ return {md5=md5}
+```
+
+Example2: write logs into the log file.
+
+```nginx
+
+ location /write_log_file {
+     default_type 'text/plain';
+
+     content_by_lua_block {
+         local ok, err = ngx.run_worker_thread("testpool", "write_log_file", "log", ngx.var.arg_str)
+         if not ok then
+             ngx.say(ok, " : ", err)
+             return
+         end
+         ngx.say(ok)
+     }
+ }
+```
+
+`write_log_file.lua`
+
+```lua
+
+ local function log(str)
+     local file, err = io.open("/tmp/tmp.log", "a")
+     if not file then
+         return false, err
+     end
+     file:write(str)
+     file:flush()
+     file:close()
+     return true
+ end
+ return {log=log}
+```
+
+[Back to TOC](#nginx-api-for-lua)
+
 Obsolete Sections
 =================
 

config

@@ -296,6 +296,7 @@ HTTP_LUA_SRCS=" \
             $ngx_addon_dir/src/ngx_http_lua_log_ringbuf.c \
             $ngx_addon_dir/src/ngx_http_lua_input_filters.c \
             $ngx_addon_dir/src/ngx_http_lua_pipe.c \
+            $ngx_addon_dir/src/ngx_http_lua_worker_thread.c \
             "
 
 HTTP_LUA_DEPS=" \
@@ -355,6 +356,7 @@ HTTP_LUA_DEPS=" \
             $ngx_addon_dir/src/ngx_http_lua_log_ringbuf.h \
             $ngx_addon_dir/src/ngx_http_lua_input_filters.h \
             $ngx_addon_dir/src/ngx_http_lua_pipe.h \
+            $ngx_addon_dir/src/ngx_http_lua_worker_thread.h \
             "
 
 # ----------------------------------------

doc/HttpLuaModule.wiki

@@ -2824,6 +2824,22 @@ This allows Lua I/O primitives to not be interrupted by Nginx's handling of vari
 
 This directive was first introduced in the <code>v0.10.14</code> release.
 
+== lua_worker_thread_vm_pool_size ==
+
+'''syntax:''' ''lua_worker_thread_vm_pool_size <size>''
+
+'''default:''' ''lua_worker_thread_vm_pool_size 100''
+
+'''context:''' ''http''
+
+Specifies the size limit of the Lua VM pool (default 100) that will be used in the [ngx.run_worker_thread](#ngxrun_worker_thread) API.
+
+Also, it is not allowed to create Lua VMs that exceeds the pool size limit.
+
+The Lua VM in the VM pool is used to execute Lua code in seperate thread.
+
+The pool is global at Nginx worker level. And it is used to reuse Lua VMs between requests.
+
 = Nginx API for Lua =
 
 <!-- inline-toc -->
@@ -7671,6 +7687,126 @@ This API was first usable in the context of [[#init_by_lua|init_by_lua*]] since
 
 This API was first enabled in the <code>v0.6.0</code> release.
 
+== ngx.run_worker_thread ==
+
+'''syntax:''' ''ok, res1, res2, ... = ngx.run_worker_thread(threadpool, module_name, func_name, arg1, arg2, ...)''
+
+'''context:''' ''rewrite_by_lua*, access_by_lua*, content_by_lua*''
+
+'''This API is still experimental and may change in the future without notice.'''
+
+'''This API is available only for Linux.'''
+
+Wrap the [http://nginx.org/en/docs/dev/development_guide.html#threads nginx worker thread] to execute lua function. The caller coroutine would yield until the function returns.
+
+Note that no ngx_lua API can be used in the `function_name` function of the `module` module since it is invoked in a separate thread.
+
+The first argument `threadpool` specifies the Nginx thread pool name defined by [thread_pool](https://nginx.org/en/docs/ngx_core_module.html#thread_pool).
+
+The second argument <code>module_name</code> specifies the lua module name to execute in the worker thread, which would return a lua table. The module must be inside the package path, e.g.
+
+<geshi lang="nginx">
+lua_package_path '/opt/openresty/?.lua;;';
+</geshi>
+
+The third argument <code>func_name</code> specifies the function field in the module table as the second argument.
+
+The type of <code>arg</code>s must be one of type below:
+
+* boolean
+* number
+* string
+* nil
+* table (the table may be recursive, and contains members of types above.)
+
+The <code>ok</code> is in boolean type, which indicate the C land error (failed to get thread from thread pool, pcall the module function failed, .etc). If <code>ok</code> is <code>false</code>, the <code>res1</code> is the error string.
+
+The return values (res1, ...) are returned by invocation of the module function. Normally, the <code>res1</code> should be in boolean type, so that the caller could inspect the error.
+
+This API is useful when you need to execute the below types of tasks:
+
+* CPU bound task, e.g. do md5 calculation
+* File I/O task
+* Call <code>os.execute()</code> or blocking C API via <code>ffi</code>
+* Call external Lua library not based on cosocket or nginx
+
+Example1: do md5 calculation.
+
+<geshi lang="nginx">
+location /calc_md5 {
+    default_type 'text/plain';
+
+    content_by_lua_block {
+        local ok, ret, md5_or_err = ngx.run_worker_thread("testpool", "calc_md5", "md5", ngx.var.arg_str)
+        if not ok then
+            ngx.say(ret)
+            return
+        end
+        if not ret then
+            ngx.say(md5_or_err)
+            return
+        end
+        ngx.say(md5_or_err)
+    }
+}
+</geshi>
+
+<code>calc_md5.lua</code>
+
+<geshi lang="lua">
+local resty_md5 = require "resty.md5"
+local resty_str = require "resty.string"
+
+local function md5(str)
+    local md5 = resty_md5:new()
+    if not md5 then
+        return false, "md5 new error"
+    end
+
+    local ok = md5:update(str)
+    if not ok then
+        return false, "md5 update error"
+    end
+
+    local digest = md5:final()
+    return true, resty_str.to_hex(digest)
+end
+return {md5=md5}
+</geshi>
+
+Example2: write logs into the log file.
+
+<geshi lang="nginx">
+location /write_log_file {
+    default_type 'text/plain';
+
+    content_by_lua_block {
+        local ok, err = ngx.run_worker_thread("testpool", "write_log_file", "log", ngx.var.arg_str)
+        if not ok then
+            ngx.say(ok, " : ", err)
+            return
+        end
+        ngx.say(ok)
+    }
+}
+</geshi>
+
+<code>write_log_file.lua</code>
+
+<geshi lang="lua">
+local function log(str)
+    local file, err = io.open("/tmp/tmp.log", "a")
+    if not file then
+        return false, err
+    end
+    file:write(str)
+    file:flush()
+    file:close()
+    return true
+end
+return {log=log}
+</geshi>
+
 = Obsolete Sections =
 
 This section is just holding obsolete documentation sections that have been either renamed or removed so that existing links over the web are still valid.

src/ngx_http_lua_common.h

@@ -288,6 +288,8 @@ struct ngx_http_lua_main_conf_s {
     ngx_queue_t          free_lua_threads;  /* of ngx_http_lua_thread_ref_t */
     ngx_queue_t          cached_lua_threads;  /* of ngx_http_lua_thread_ref_t */
 
+    ngx_uint_t           worker_thread_vm_pool_size;
+
     unsigned             requires_header_filter:1;
     unsigned             requires_body_filter:1;
     unsigned             requires_capture_filter:1;

src/ngx_http_lua_module.c

@@ -646,6 +646,13 @@ static ngx_command_t ngx_http_lua_cmds[] = {
       0,
       NULL },
 
+    { ngx_string("lua_worker_thread_vm_pool_size"),
+      NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1,
+      ngx_conf_set_num_slot,
+      NGX_HTTP_MAIN_CONF_OFFSET,
+      offsetof(ngx_http_lua_main_conf_t, worker_thread_vm_pool_size),
+      NULL },
+
     ngx_null_command
 };
 
@@ -975,6 +982,8 @@ ngx_http_lua_create_main_conf(ngx_conf_t *cf)
         return NULL;
     }
 
+    lmcf->worker_thread_vm_pool_size = NGX_CONF_UNSET;
+
     dd("nginx Lua module main config structure initialized!");
 
     return lmcf;
@@ -1058,6 +1067,10 @@ ngx_http_lua_init_main_conf(ngx_conf_t *cf, void *conf)
     }
 #endif
 
+    if (lmcf->worker_thread_vm_pool_size == NGX_CONF_UNSET_UINT) {
+        lmcf->worker_thread_vm_pool_size = 100;
+    }
+
     return NGX_CONF_OK;
 }