gng.conf − Gigapxy engine configuration file.

gng(1) is the gigapxy(1) data processing daemon, responsible for the I/O on behalf of user requests relayed to it by gws(1). The configuration file contains the parameters read by the daemon at the launch. The file is in easy−to−read libconfig format. An example of a gng.conf is provided with the installation.

Once the parameters are read by gng, the daemon operates with those values util the configuration is re−loaded in response to SIGHUP.

All gng(1) settings begin with the ng. prefix, as in ng.section.param. A configuration file may also contain settings intended for other modules; gng would ignore all non−gng settings.

The configuration settings are listed below. The default value for a setting is given in square brackets as [default]. Parameters without default values are mandatory.

ng.ng_socket_path = path [/var/run/gpx-ngcomm.socket]
is the domain socket path for communications between gws and the attached gng’s.

ng.log.level_default = err| crit| warn| norm| info| debug [info]
Defines the level of verbosity for the log.

ng.log.file = path
Full path to log.

ng.log.max_size_mb = num [16]
Maximum file size (in Mb, i.e. 1048576−byte chunks). Log is rotated when this size is exceeded. gng will force−rotate its current log in response to SIGUSR1.

ng.log.max_files = num [16]
Maximum number of files to rotate to. The next rotation after this limit removes the oldest rotated log.

ng.log.time_format = local| gmt| raw| raw_mono| no_time| [local]
Sets format to display timestamps for log entries. local will loglocal−timezone specific time in YYYY-MM-DD HH24:MI TZ format. gmt will log GMT time in the same human−readable format as local; raw logs high−resolution time as the number of seconds.nanoseconds since the Epoch (1970-01-01 00:00:00 UTC); raw−mono logs system−specific monotonic time (used for timespan measurement, not correlated to clock time). no_time logs no time at all.

ng.log.show_pid = true|false [true]
Display PID as a log entry field.

ng.log.enable_syslog = true|false [true]
Write errors, warnings and critical messages to syslog(2). = dirname [/var/run/gigapxy]
Directory for the pidfile (must be writable by ng.run_as_user). = filename [gng−{user_port}.pid]
Name (w/o directory part of the path) of the pidfile, the default value uses the user−request listener port number.

ng.idle_clk_ms = milliseconds [−1]
Time (ms) to wait before doing any idle-time tasks, −1 = no limit. This sets the resolution (or granularity) for the timeouts or any other tasks done in idle time. The default value (−1) will have it perform idle tasks only when an actual event (connection, signal, etc.) interrupts the wait loop. gng will set the idle clock to the minimum value of a channel/client timeout.

ng.run_as_user = username []
Run as this user when running as a daemon (if empty, do not switch).

ng.run_as_uid = uid [−1]
Run as the given user (uid) when running as a daemon (if −1, do not switch). If gid is not specified, then gid = uid. uid > 0 will override run_as_user.

ng.run_as_gid = gid [−1]
Run in the given group (gid) when running as a daemon (if −1, gid = uid).

ng.non_daemon = true | false [false]
If started as root, become a daemon if true.

ng.enforce_core_dumps = true | false [false]
When set to true, the process invokes the necessary syscalls to make itself core-dumpable and set core limit to unlimited. The default value of false leaves it to the shell defaults. NB: Under certain Linux versions, UID−changing daemons become non−core−dumpable (see /proc/sys/fs/suid_dumpable and prctl(2) for details).

ng.no_rtp_strip = true | false [false]
When set to true, the engine does not attempt to convert RTP-over-TS into plain TS datagrams (enabled by default). When ’stripping’ is disabled, gng would consider RTP packets as non-TS and relay them AS-IS.

IMPORTANT: data buffers MUST be memory−mapped (mmap_files or mmap_anon) for gng to
perform RTP stripping. Disable RTP stripping (ng.no_rtp_strip = true) if using non−memory buffers.

ng.use_sendfile = true | false [true on FreeBSD otherwise false]
Prefer to use sendfile(2) to send out data. This makes a big difference on FreeBSD, which implements zero-copy through this syscall. Setting this to true on Linux may or may not improve performance (so it’s false by defaylt under Linux).

ng.quiet = true | false [false]
No output to stdout/stderr if true.

ng.cpunum = -1 | 0 .. N [-1]
Set affinity to CPU #N (zero-based) for this process, unless -1 (or <0).

ng.process_limits.rss = {N}{suffix} ["0"]
Resident memory cap: a process cannot exceed this amount in resident memory, memory allocation call(s) should fail. NB: This limit cannot be enforced under Linux, where it would be replaced by RLIMIT_AS (virtual memory cap). If both RSS and VMEM are to be limited under Linux, the smaller value is used with RLIMIT_AS. Under FreeBSD, RSS limit is fully supported.

ng.process_limits.vmem = {N}{suffix} ["0"]
Virtual memory cap = RLIMIT_AS. Used in place of RSS cap under Linux. Both Linux and FreeBSD fully support it.

ng.max_channels = num [200]
Maximum number of channels allowed (per engine).

ng.max_channel_clients = num [500]
Maximum number of clients per single channel.

ng.channel_io_timeout_sec = seconds [5]
Maximum time (in seconds) to wait on I/O for a channel.

ng.client_io_timeout_sec = seconds [5]
Maximum time to wait on I/O for a client.

ng.client_busy_timeout_sec = seconds [86400 = 24 hours]
Maximum time for a client session.

ng.can_extend_clients = true|false [false]
If a client times out, check if there’s pending (channel) data and the client is writable. If writable, extend its wait period (just this one time) by client_io_timeout_sec.

ng.client_socket_sndbuf_size = bytes [system default]
Client (sending) socket send buffer size (bytes).

ng.channel_socket_rcvbuf_size = bytes [system default]
Channel (receiving) socket buffer size (bytes).

ng.channel_lo_wmark = bytes, 0 = none [0]
Low watermark for channel sockets.

ng.client_tcp_cork = true|false [false]
Use Linux TCP_CORK socket option to aggregate client packets. Linux only.

ng.client_tcp_nopush = true|false [false]
Use BSD TCP_NOPUSH socket option to aggregate client packets. BSD only.

ng.multicast_ttl = hops [2]
Multicast TTL value set for the outgoing mulitcast traffic.

The following section specifies the parameters needed for engines to report traffic throughput statistics, queried using report admin request. See gigapxy(1) for details on reports and admin request particulars.

ng.tput_stats.enabled = true | false [true]
Do not provide channel/client storage unless true. Please note that engines will use additional CPU cycles to gather and calculate relevant statistics.

ng.tput_stats.channel_path = posix_shmem_path [/gxy-cha.shm]
POSIX shared memory path for channel storage (<= 32 characters). Note: should match the corresponding gws setting.

ng.tput_stats.client_path = posix_shmem_path [/gxy-cli.shm]
POSIX shared memory path for client storage (<= 32 characters). Note: should match the corresponding gws setting.

ng.tput_stats.channel_report_ms = milliseconds [5000]
Report channel throughput every N milliseconds. (Will save the statistics in shared memory.)

ng.tput_stats.client_report_ms = milliseconds [5000]
Report channel throughput every N milliseconds. (Will save the statistics in shared memory.)

ng.tput_stats.max_packet_delta = bytes [−1]
Warn if two consecutive packets differ by more that N bytes, −1 = ignore. This setting allows to watch out for inconsistencies in the UDP streams, where all messages are supposed to be of the same size.

Attempt N reconnects with gws, unlimited if −1, none if 0. If the controlling gws crashes, gng makes a number of attemtps, separated by pauses, to re−attach to it. Therefore, if a monitor on the crashed gws restarts it successfully, the formerly−attached gng’s may re−attach. = milliseconds [500]
Delay (in milliseconds) between reconnect attempts.

ng.http_data_content_type = type_specifier [application/octet-stream]
HTTP Content-Type for data payload.

The following section specifies the parameters for the internal cache used by gng to multiplex access to channel data. For each channel (that needs to be cached) gng maintains a chain of buffers, representing consecutive segments for traffic data.

ng.bufd.keep_files = true | false [false]
Do not unlink(2) bufd files (make them visible). This is a debugging option.


ng.bufd.mmap_files = true | false [true]
Map bufd files into memory. Results in faster access to cache but may exhaust host memory.

ng.bufd.mmap_anon = true | false [false]
Allocate buffers in memory w/o using any filesystem space (i.e. buffers are not backed up by files). This option provides the fastest access to cache but is limited by process’s memory constraints. It also overrides mmap_files.

MUST USE mmap_anon OR mmap_files if you intend to strip RTP datagrams.
can only strip RTP datagrams in memory, so memory mapping is a MUST if you’re handling RTP traffic. If you’re NOT handling RTP and economize on RAM using file-backed buffers, then please disable RTP stripping by setting ng.no_rtp_strip = true

ng.bufd.mlock = true | false [false]
mlock(2) data buffers into physical memory. Make sure your system parameters allow this, for reference see mlock(2) manpage.

ng.bufd.data_dir = pathname [/tmp]
Directory to place bufd files into.

The following three settings affect the way gng caches data. There is a certain amount that can be kept per channel to ensure that new clients can start receiving data without delay. The settings below regulate that amount and set the point (in the cache) from which data gets served to a new client.

ng.bufd.min_total_duration_sec = seconds [5]
Minimum of data cached for a channel, measured in time it took to receive it. No channel buffers get recycled until this much data has been saved. The exact amount preserved in cache could be a above but never below the imposed threshold; gng would recycle a buffer only if, after its removal, the cache would still have >= seconds worth of data.

ng.bufd.min_total_size = bytes [1048576]
Minimum of data cached for a channel, in bytes. No channel buffers get recycled until this much data has been saved.

The two settings above work in tandem, each of them setting a threshold. gng will consider that enough data has been cached as soon as either or both of those thresholds have been reached: if, for instance, the first setting is 5 seconds and the second one is 10485760 bytes (10 Mb), then enough is as soon as we’ve cached 10Mb or accumulated more than 5 seconds worth of data (if the channel is slow, it may be less than 10Mb).

Channel data is stored as a sequence of buffers, from the most−recently−received one − the HEAD, to the oldest one − the TAIL.

A newly−joined client/subscriber needs the first data buffer to start with. The setting below defines the algorithm gng would use to pick one.

ng.bufd.start_mode = 0 | 1 | −1 | 2 [1]
Defines the method to pick the initial buffer for a client:

0 (HEAD) picks the most−recently−received buffer, offset: 0 (cached: all of the most recent buffer).

2 (EDGE) picks the most−recently−received buffer, offset: END−OF−DATA (no data cached).

1 (MIN_CACHED) picks the buffer that allows to transfer N seconds or M bytes of data, offset: 0 (cached: N seconds/M bytes).

−1 (TAIL) picks the oldest buffer in cache, offset: 0 (cached: all current data for the channel).

All the above methods, except EDGE (2) position a new client at zero offset in the selected buffer. This way there is always some data to send to the client right away, without I/O wait. EDGE ensures a no-cache policy: only the data received during the client’s lifespan gets relayed to the destination.

MIN_CACHED (1) uses ng.bufd.min_total_duration_sec and ng.bufd.min_total_size to determine which buffer to pick. The algorithm starts at the HEAD and moves towards the tail accumulating the volume and time for each buffer it lands on; as soon as either of the thresholds is reached, the buffer is selected as the one to start at.

ng.bufd.burst_mode = 0 (none) | 1 (scan) | 2 (burst) [1]
Defines the method used to prevent excessive growth of buffer chains using a ’bubble-burst’ technique. A ’bubble’ is a (long) sequence of unused buffers (U) squeezed in between a small number of active buffers (A). A slow client may claim (lock on) a buffer and then slow down, while other clients go ahead. The tail-side buffer would be still locked while the buffers towards the head get used and un−locked: AAUUUUUUUUUUUA. The U-sequence is the ’bubble’. In mode 1 (scan) gng scans a channel looking for a bubble (and warns if it finds one), in mode 2 2 (burst) it also tries to invalidate the rightmost A−buffer and release the underlying U−sequence (the bubble).

ng.bufd.max_unit_count = num [128]
Maximum number of buffers for all channels within the given gng instance. Not all buffers, as a rule, get allocated at once. This sets the limit to the number of buffers across all channels.

ng.bufd.prealloc_count = buffers [max_unit_count / 4]
Number of shared buffers to pre-allocate at the engine’s start.

ng.bufd.max_units_per_channel = num [1/3 of max_unit_count, yet within 4..12 range]
Maximum numbers of buffers per channel. Setting this to a well-balanced value will provide for fair distrubution of buffers across channels and will prevent hogging buffer space by channels with slow clients.

ng.bufd.max_unit_size = bytes [16777216]
Maximum number of bytes in a single buffer. When this threshold is hit, another buffer is added to cache.

ng.bufd.max_dgram_size = bytes [1500]
Maximum size of a (UDP) datagram expected (should not exceed MTU). NB: must be adjusted if using jumbo frames.

ng.bufd.max_unit_duration_sec = seconds [30]
Maximum duration (seconds) of a single buffer. When this threshold is hit, another buffer is added to cache.

ng.bufd.allow_emergency_recycle = true | false [false]
Allow to force−recycle the oldest buffer when cannot allocate a new one.

ng.bufd.client_catch_up_ms = ms [0]
Advance to the most recent buffer if no data gets sent out within the given time period. Once a client times out, its output marker is shifted to ’catch up’ with the source data flow. For start_mode = EDGE (2), the offset would be at the end of the head buffer (edge), for other modes - byte 0 of the head buffer.

associated with a channel to track incoming datagrams. Certain features rely heavily on such tracking. A datagram index holds a number of datagram sequences. Each sequence accounts for a number of consecutive datagrams of the same size.

ng.bufd.stale_dgram_ms = ms [0]
If non−zero, the value sets expiry period for inbound data in UDP channels. If a client finds itself at the point of needing to transmit a stale datagram, its output offset gets shifted to byte 0 of the most−recently−received datagram.

ng.bufd.max_dseq_ms = ms [8 * stale_dgram_ms]
Maximum time in milliseconds (worth of data) to aggregate in a single datagram sequence. This setting could be made smaller to increase precision of time measurement for stale_dgram_ms or made larger to aggregate more datagrams in a single sequence (in order to save space).


ng.bufd.recycle_timeout_sec = seconds [30]
Time period to check for stale channel buffers that could be recycled (−1|0 = never).

ng.transfer_buffer_size = bytes [1048576]
Size of the intermediate buffer used to facilitate I/O. This setting is unused when cache buffers are memory−mapped.

This section regulates delaying data output to reduce the number of write(2) syscalls. Data is accumulated until the saved portion is large enough.

ng.cli_write_delay.enabled = true|false [true]
Write delays are enabled if set to true.

ng.cli_write_delay.timeout_ms = delay_ms [100]
Delay for no more than N milliseconds.

ng.cli_write_delay.max_buffered = max_bytes [1048576]
Delay up to max_bytes of data, disregard if 0.

Performance sensors allow to measure resource utilization between two specific points within the application, using the metrics provided by utime(2) utime(2) call at each end of the sensor. All sensor data will be printed out at the application exit in the format similar to the output of time(1) utility.

Performance sensors are a debugging/profiling facility and incur additional load on the system.


Defined sensors:
= application runtime; ev_loop = event processing (all events); ev_read = reading/processing inbound data; ev_write = writing/processing outbound data; ev_err = processing error events; ev_pp = post−processing events; ng_chaio = channel data I/O; ng_clio = client data I/O.

ng.psensors.enable_all = true|false [false]
Enables all sensors if true, disables all otherwise. This is to initialize the set of enabled−sensor flags to either all ones (if enabled) or all zeros. This setting is to be used in combination with ng.psensors.except.

ng.psensors.except = sensor_list []
Enables sensors in the list if ng.psensors.enable_all is true, or disables those sensors if false. This way enable_all is used to initialize the set of sensors while except narrows it down by enabling/disabling its specific elements.


ng.psensors.enable_all = true; # Enable all sensors.

ws.psensors.except = ["ev_read", "ev_write"]; # Disable those listed herein.

Enables all sensors except ev_read and ev_write.


ng.psensors.enable_all = false; # Disable all sensors.

ng.psensors.except = ["ev_read", "ev_write"]; # Enable those listed herein.

Enables ev_read and ev_write sensors, all others are disabled.


Pavel V. Cherenkov