Skip to main content

Datometry Documentation

Hyper-Q System Log File

The Hyper-Q system log file contains metrics about the operational status of Hyper-Q. Hyper-Q records the collected metrics to the file at approximately five second intervals, and each live in the file is a snapshot of the metrics at the indicated time.

The system log is a comma-separated values (CSV) file, a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas.

Table 17. Hyper-Q System Log Columns

Column Name

Description

timestamp

The timestamp when the event occurred, using the local server timezone and formatted with millisecond-precision timestamps as: YYYY-MM-DD HH:MM:SS.mmm

session_count

The number of active client sessions connecting to Hyper-Q.

lsn_count

The number of concurrent bulk import and export sessions.

process_count

The number of running Erlang processes.

port_count

The number of open network sockets, files, and external processes.

cpu_util

Percentage of CPU utilization as reported by the operating system kernel for all operating system processes.

mem_util

Percentage of memory utilization of all operating system processes on the host system.

reductions

Amount of computation performed since the last entry.

io_input

The number of bytes read from network sockets since the last entry.

io_output

The number of bytes written to network sockets since the last entry.

sds_file_writer_out

The number of bytes written to bulk import staging files since the last entry.

sds_result_device_out

The the total size in bytes written to result spill files since the last entry. Spill files save intermediate results when there is insufficient memory to execute a query in memory. Disk I/O is much slower than memory access, so a query that creates spill files will take longer to complete.

mem_alloc

Total memory allocated to Erlang carriers. A carrier is a portion of memory allocated from either the operating system or a super carrier (see note). There are two types of carriers: single-block carriers which only contain one large block, and multi-block carriers which contain multiple blocks.

To learn more about memory allocation in Erlang, see the following resources:

Note

The mem_alloc metric does not count super carriers. A super carrier is a large memory area allocated when starting the Erlang virtual machine that can be used during runtime to allocate memory to normal carriers.

To learn more about super carriers, see Super Carrier in the Erlang documentation.

total

The total amount of memory in bytes allocated to Hyper-Q.

processes

The total amount of memory in bytes allocated to Erlang processes.

processes_used

The total amount of memory in bytes currently in use by Erlang processes.

system

The total amount of system overhead memory in bytes.

atom

The total amount of memory in bytes allocated to the atom table. This is a portion of the system memory.

An atom table is a system-defined table that stores strings and corresponding identifiers. An application places a string in an atom table and receives a 16-bit integer, called an atom, that can be used to access the string.

To learn more about atom tables and memory usage, see Memory in the the Erlang documentation.

atom_used

The total amount of memory in bytes used by atoms in the atom table. This is a portion of the system memory.

binary

The total amount of memory in bytes used for binary data. This is a portion of the system memory.

code

The total amount of memory in bytes used for executable code. This is a portion of the system memory.

ets

The total amount of memory in bytes used for Erlang Term Storage (ETS) tables. This is a portion of the system memory.

ETS can store large amounts of data and provides constant time data access. Tables in ETS are created and owned by individual processes. When an owner process terminates, its tables are destroyed. You can have as many ETS table as you want, the only limit is the server memory.

To learn more about ETS tables, see ETS and Memory in the Erlang documentation.