dlh: Top-level Files of tip

Files in the top-level directory from the latest check-in

example
src
build.conf.lua
build_and_test_dlh.sh
Changes.txt
dlh_cqueues.lua
dlh_luaposix.lua
dlh_luasocket.lua
dlh_osn.lua
dlh_osn.perf.lua
license.txt
Readme.txt
dlh - dynamic lua hypertext
===========================
dlh is a web application server for Lua5.4 (5.3)
dlh is designed to run behind a http server (proxy) to run Lua scripts for dynamic content.
(nginx, apache, lighttpd, ... are more suitable and performant for static files)
dlh is one executable file, that depends only on: osn (recommended), luaposix, luasocket or cqueues
-> osn -> https://holmeinbuch.de/repo/osn/

Goals:
- creating dynamic content for the web (php, java servlet -> Lua)
- KISS - "no" dependencies, single executable (It is still Lua -> requires Lua and socket lib)
- simple routing
- cached dlh, dynamic (hot) reload
- "session" cache for dlh
- simple stat / control interface
- providing html headers and (form) data to dlh
- save dlh environment: if dlh crashes, server is stable

NON-Goals:
- ssl (done by webserver / proxy)
- zip (done by webserver / proxy)
- static content delivery (done by webserver / proxy)
- streaming
- http/2.0
- no integrated storage backend (use your (sql, kv, nosql, ...) modules)
- no multithreading ... start multiple instances (there is detach)


Running a server
----------------
> lua dlh [config]
dlh = _dlh_(osn|luaposix|luasocket|cqueues).lua (or build your own)
Default config is /etc/dlh.conf.lua


Configuration
-------------
The configuration file contains:
- a server section to define [ip,]port/usock
- a log section to define log function calls (there is no special logger implemented, use whatever logger you like)
- a request_map path mapping to dlh files (file name or table with local config)
The dlh files will inherit the global configuration, but can be replaced by local definitions
The example/dlh.conf.lua contains a detailed description and examples


Creating a dlh
--------------
A dlh file simply contains Lua code.
The server will load the dlh file and execute it in a save environment.
(it will be lazy loaded once on the first call -> mode for preload/unload available)
It provides special tables and functions and "disables" some Lua functions.
+ request table: contains data for this request
+ server table: contains environment and helper functions
+ response table: contains functions to create the response
  -> for more details see: example/example_browse.dlh (just a feature demo, not save for use)
+ detach function: will detach the current connection from the server:
  -> if the template is fast, don't use detach
  -> detach costs time, but after the call it will not block the server
  -> after the detach call, modifications of vars in (dlh_cache / share) will be lost
- assert, collectgarbage, error -> will log an error, not allowed
- print an io.write will write to log.error
- io.{close, flush, input, lines, output, read, stdout, stderr, stdin} -> will log an error, not allowed
- os.exit -> will log an error, not allowed

The server provides special vars:
server.{
  dlh_config -> This var can be set in the dlh.conf.lua, there is no type specified, but this config
                setting is available in the dlh (could be an ID, config filename to load, config table, ..)
  dlh_cache -> This is a cache only available for that dlh
  share -> This is a cache shared by the whole server and could be used to exchange data between dlh
}

There are special admin calls in the server table, which allow to modify dlh mode and server.
These require special mode dlh_admin and/or server_admin set. (Only available to set in config file)
-> there is a template: example/server_control.dlh which provides an webinterface / "api" calls
The dlh_cache and server share are for caching session data, but will NOT survive a server restart.
Make sure, your dlh works with an empty cache...
Hints:
  - a dlh can be loaded in a "uncache" mode -> dlh_cache is always empty in that case.
  - there is also the "unload" mode, which releases the dlh template and will reload
    from disk on every request -> for easy development


tool:
There are builtin tools (from tool repo)
server.{
  readConfig -> used by the server to read the config file into a table
                might be interesting for dlh too: https://holmeinbuch.de/repo/tool/file?name=tool.readConfig.lua&ci=tip
  parseTmpl() -> for template parsing -> https://holmeinbuch.de/repo/tool/file?name=tool.parseTmpl.lua&ci=tip
                see: example/example_template.[dlh|html] (this is an optional build feature, which is not required in custom build)
  urlEncode / urlDecode -> https://holmeinbuch.de/repo/tool/file?name=tool.urlXxcode.lua&ci=tip
                string in string out... (the server uses urlDecode on all args and simple data)
}


Management web interface
------------------------
The server_control.lua is providing a simple management web interface.
It can be mapped like any other dlh (but shouldn't be open to public access)
For full access it needs the dlh mode set to "dlh_admin,server_admin"
curl and wget will return some text statistic and can do "api" calls
with other browsers (text or ui) there are some more control options in html
to browse through and simply change settings.


Building a server executable
----------------------------
There are already prebuilds: dlh_(osn|luaposix|luasocket|cqueues).lua
Details, about what is included is in the build.conf.lua

The source code uses different files with code snippets (and tests)
Requirements to build the binary:
- build -> https://holmeinbuch.de/repo/build/ (the build program)
- tool -> https://holmeinbuch.de/repo/tool/ (common snippets)
dlh expects "tool" next to it's dir ("../tool/") or as a sub dir ("tool/")


Performance
-----------
There is no such thing as x requests per second...
It is a different performance, if I use an empty dlh which writes "Hello, World!" or
a dlh, that is doing a lot of disk and/or database operations.
(e.g. https://suade.org/12-requests-per-second-with-python/ )

The server is a single thread application server. In my opinion all parallel stuff
adds a lot of complexity and problems. Erlang has a good approach on that, for really
a lot of independent workers, that might be a better choice.
The time a dlh needs to execute is the time others are waiting.
If higher performance is needed, multiple instances can be started and proxied.
(Will add complexity on session data)
In a single thread application there exists a simple way to "crash" the server.
Call a dlh which just contains a endless loop:

while true do nothing = true end

There is no way for the server process to:
- know how long a dlh should run
- kill a dlh, that is gone rogue (just one process)

A workaround can be to monitor a simple dlh and restart the server, if there is no
response in time. But just don't do experiments in a productive environment.
There is an example monitor script in example/monitor_dlh.sh

I did test different options, dlh can be build as SCGI or FCGI server instead of http
...but in the end the perfomance will not get better with these builds.
-> prefer unix-socket over tcp port

Caching can improve request times (dlh works with caches), but it will also get more complex.

Running behind a (nginx) proxy will help, that slow uploads are covered by the proxy.
dlh will get the upload more fast and not block for a long time - check out "tmpname_prefix",
so you don't have to copy the file again to another filesystem.
For sending try to use the proxy as well:
response.setHeader("X-Accel-Redirect", "/path/to/file")
return

X-Sendfile:
- nginx: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
- lighttpd: https://redmine.lighttpd.net/projects/lighttpd/wiki/X-LIGHTTPD-send-file


Source files
------------
build.conf.lua
  configuration file for builds
build_and_test_dlh.sh
  script to test builds and build the executables
dlh.conf.lua
  config manual, a config file with every option and instruction
src/dlh._main.lua
  the main file invoked by build script, contains the default setup and frame.
  It will use one of the runServer invocations:
  - src/dlh.runServer_osn.lua
  - src/dlh.runServer_luaposix.lua
  - src/dlh.runServer_luasocket.lua
  - src/dlh.runServer_cqueues.lua
  - src/dlh.runServer_fcgi.lua

  - src/dlh.parseConfig.lua
    read and parse the config file
    - src/dlh.getRequestMap.lua
      parser for the request_map in the config file
    - src/dlh.getRequestTree.lua
      build request tree for fast routing access
    - src/dlh.loadDlh.lua
      (pre)load the dlh file

  - src/dlh.parseRequest.lua
    initializes the request table by parsing the input stream (method, http_version, remote_up, home, header table, cookie, arg table, data table)
    - src/dlh.readRequest.lua
      low level socket stream input reading (logic)
    - src/dlh.parseRequestLine.lua
    - src/dlh.parseRequestHeader.lua
    - src/dlh.parseRequestHeaderSCGI.lua
    - src/dlh.parseRequestHeaderFCGI.lua
    - src/dlh.parseRequestArg.lua
    - src/dlh.parseRequestBody.lua
      supported body type:
        - application/x-www-form-urlencoded
        - multipart/form-data
        - application/octet-stream
        - application/json -> application/octet-stream (body is accepted own json parser needs to be invoked)
    - src/dlh.parseQueryString.lua

  - src/dlh.writeResponse.lua
      initializes the response, calls the dlh and writes to the output stream
    - src/dlh.initDlhResponse.lua
      initializes the response table (functions: getStatus, setStatus, setHeader, setCookie and write)
    - src/dlh.initDlhServerInterface.lua
      initializes the server table
    - src/dlh.getSaveEnv.lua
      creates a save environment table, that is used for dlh execution
    - src/dlh.writeResponseHeader.lua
      takes care of headers (HTTP_STATUS_CODE, date, content-length(, location))
src/fcgi.c
  simple fcgi mapper to use in Lua


Projects from other developers
==============================
webserver
---------
makoserver (amazing project !not opensource):
  https://makoserver.net/
   - https://fuguhub.com/
   - https://realtimelogic.com/products/barracuda-application-server/ (library)
openresty (nginx module - luajit):
  https://openresty.org/en/
nginx-lws (nginx module - Lua5.X):
https://github.com/anaef/nginx-lws
pegasus:
  https://evandrolg.github.io/pegasus.lua/
Xavante:
  https://keplerproject.github.io/xavante/index.html
ladle:
  https://github.com/danielrempel/ladle
lua-httpd:
  https://steve.fi/software/lua/lua-httpd/
hinsightd/hin9 (io_uring webserver with Lua config):
  https://tiotags.gitlab.io/hinsightd/
  https://codeberg.org/tiotags/hin9
lwan.ws
  https://github.com/lpereira/lwan
lwan-lua
  https://github.com/withinstem/lwan-lua

framework
---------
lapis:
  https://leafo.net/lapis/
sailor:
  https://github.com/sailorproject/sailor
tulip:
  https://git.sr.ht/~mna/tulip
turbo (luajit 2):
  https://github.com/kernelsauce/turbo
luvit (asynchronous i/o - create your own webserver):
  https://luvit.io

cgi / lib
---------
haserl (cgi in bash or lua)
  http://haserl.sourceforge.net/index.html
lua-http:
  https://github.com/daurnimator/lua-http

go + lua
--------
heart (luajit):
  https://github.com/Hyperspace-Logistics/heart
algernon:
  https://algernon.roboticoverlords.org/
Luax -> JSX for Lua, go server with Lua templating:
  https://bvisness.me/luax/

static website
--------------
LuaWebGen:
  https://github.com/ReFreezed/LuaWebGen
Luapress:
  https://github.com/Fizzadar/Luapress

Lua-users Wiki
--------------
http://lua-users.org/wiki/LuaWebserver
( http://lua-users.org/wiki/LibrariesAndBindings )