> ## Documentation Index > Fetch the complete documentation index at: https://private-7c7dfe99-fix-nav-issues.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. > System table containing stack traces collected by the sampling query profiler. # system.trace_log **Querying in ClickHouse Cloud** The data in this system table is held locally on each node in ClickHouse Cloud. Obtaining a complete view of all data, therefore, requires the `clusterAllReplicas` function. See [here](/reference/system-tables/overview#system-tables-in-clickhouse-cloud) for further details.

Description

Contains stack traces collected by the [sampling query profiler](/concepts/features/performance/troubleshoot/sampling-query-profiler). ClickHouse creates this table when the [trace\_log](/reference/settings/server-settings/settings#trace_log) server configuration section is set. Also see settings: [query\_profiler\_real\_time\_period\_ns](/reference/settings/session-settings#query_profiler_real_time_period_ns), [query\_profiler\_cpu\_time\_period\_ns](/reference/settings/session-settings#query_profiler_cpu_time_period_ns), [memory\_profiler\_step](/reference/settings/session-settings#memory_profiler_step), [memory\_profiler\_sample\_probability](/reference/settings/session-settings#memory_profiler_sample_probability), [trace\_profile\_events](/reference/settings/session-settings#trace_profile_events). To analyze logs, use the `addressToLine`, `addressToLineWithInlines`, `addressToSymbol` and `demangle` introspection functions.

Columns

* `hostname` ([LowCardinality(String)](/reference/data-types/lowcardinality)) — Hostname of the server executing the query. * `event_date` ([Date](/reference/data-types/date)) — Date of sampling moment. * `event_time` ([DateTime](/reference/data-types/datetime)) — Timestamp of the sampling moment. * `event_time_microseconds` ([DateTime64(6)](/reference/data-types/datetime64)) — Timestamp of the sampling moment with microseconds precision. * `timestamp_ns` ([UInt64](/reference/data-types/int-uint)) — Timestamp of the sampling moment in nanoseconds. * `revision` ([UInt32](/reference/data-types/int-uint)) — ClickHouse server build revision. When connecting to the server by `clickhouse-client`, you see a string similar to `Connected to ClickHouse server version 19.18.1.`. This field contains the `revision`, but not the `version` of a server. * `trace_type` ([Enum8('Real' = 0, 'CPU' = 1, 'Memory' = 2, 'MemorySample' = 3, 'MemoryPeak' = 4, 'ProfileEvent' = 5, 'JemallocSample' = 6, 'MemoryAllocatedWithoutCheck' = 7, 'Instrumentation' = 8)](/reference/data-types/enum)) — Trace type: `Real` represents collecting stack traces by wall-clock time. `CPU` represents collecting stack traces by CPU time. `Memory` represents collecting allocations and deallocations when memory allocation exceeds the subsequent watermark. `MemorySample` represents collecting random allocations and deallocations. `MemoryPeak` represents collecting updates of peak memory usage. `ProfileEvent` represents collecting of increments of profile events. `JemallocSample` represents collecting of jemalloc samples. `MemoryAllocatedWithoutCheck` represents collection of significant allocations (>16MiB) that is done with ignoring any memory limits (for ClickHouse developers only).`Instrumentation` represents traces collected by the instrumentation performed through XRay. * `cpu_id` ([UInt64](/reference/data-types/int-uint)) — CPU identifier. * `thread_id` ([UInt64](/reference/data-types/int-uint)) — Thread identifier. * `thread_name` ([LowCardinality(String)](/reference/data-types/lowcardinality)) — Thread name. * `query_id` ([String](/reference/data-types/string)) — Query identifier that can be used to get details about a query that was running from the query\_log system table. * `trace` ([Array(UInt64)](/reference/data-types/array)) — Stack trace at the moment of sampling. Each element is a virtual memory address inside ClickHouse server process. * `size` ([Int64](/reference/data-types/int-uint)) — For trace types Memory, MemorySample, MemoryAllocatedWithoutCheck or MemoryPeak is the amount of memory allocated, for other trace types is 0. * `ptr` ([UInt64](/reference/data-types/int-uint)) — The address of the allocated chunk. * `memory_context` ([Enum8('Unknown' = -1, 'Global' = 0, 'User' = 1, 'Process' = 2, 'Thread' = 3, 'Max' = 4)](/reference/data-types/enum)) — Memory Tracker context (only for Memory/MemoryPeak): `Unknown` context is not defined for this trace\_type. `Global` represents server context. `User` represents user/merge context. `Process` represents process (i.e. query) context. `Thread` represents thread (thread of particular process) context. `Max` this is a special value means that memory tracker is not blocked (for blocked\_context column). * `memory_blocked_context` ([Enum8('Unknown' = -1, 'Global' = 0, 'User' = 1, 'Process' = 2, 'Thread' = 3, 'Max' = 4)](/reference/data-types/enum)) — Context for which memory tracker is blocked (for ClickHouse developers only): `Unknown` context is not defined for this trace\_type. `Global` represents server context. `User` represents user/merge context. `Process` represents process (i.e. query) context. `Thread` represents thread (thread of particular process) context. `Max` this is a special value means that memory tracker is not blocked (for blocked\_context column). * `event` ([LowCardinality(String)](/reference/data-types/lowcardinality)) — For trace type ProfileEvent is the name of updated profile event, for other trace types is an empty string. * `increment` ([Int64](/reference/data-types/int-uint)) — For trace type ProfileEvent is the amount of increment of profile event, for other trace types is 0. * `symbols` ([Array(LowCardinality(String))](/reference/data-types/array)) — If the symbolization is enabled, contains demangled symbol names, corresponding to the `trace`. Symbolization can be enabled or disabled in the `symbolize` setting under `trace_log` in the server configuration file. * `lines` ([Array(LowCardinality(String))](/reference/data-types/array)) — If the symbolization is enabled, contains strings with file names with line numbers, corresponding to the `trace`. * `function_id` ([Nullable(Int32)](/reference/data-types/nullable)) — For trace type Instrumentation, ID assigned to the function in xray\_instr\_map section of elf-binary. * `function_name` ([Nullable(String)](/reference/data-types/nullable)) — For trace type Instrumentation, name of the instrumented function. * `handler` ([Nullable(String)](/reference/data-types/nullable)) — For trace type Instrumentation, handler of the instrumented function. * `entry_type` ([Nullable(Enum8('Entry' = 0, 'Exit' = 1))](/reference/data-types/nullable)) — For trace type Instrumentation, entry type of the instrumented function. * `duration_nanoseconds` ([Nullable(UInt64)](/reference/data-types/nullable)) — For trace type Instrumentation, time the function was running for in nanoseconds. **Aliases:** * `build_id` — Alias for `'E2EEB60ED9503BFFA825B628D480CCDC40B2D416'`. The symbolization can be enabled or disabled in the `symbolize` under `trace_log` in the server's configuration file.

Example

```sql theme={null} SELECT * FROM system.trace_log LIMIT 1 \G ``` ```text theme={null} Row 1: ────── hostname: clickhouse.eu-central1.internal event_date: 2025-11-11 event_time: 2025-11-11 11:53:59 event_time_microseconds: 2025-11-11 11:53:59.128333 timestamp_ns: 1762862039128333000 revision: 54504 trace_type: Instrumentation cpu_id: 19 thread_id: 3166432 -- 3.17 million query_id: ef462508-e189-4ea2-b231-4489506728e8 trace: [350594916,447733712,447742095,447727324,447726659,221642873,450882315,451852359,451905441,451885554,512404306,512509092,612861767,612863269,612466367,612455825,137631896259267,137631896856768] size: 0 ptr: 0 memory_context: Unknown memory_blocked_context: Unknown event: increment: 0 symbols: ['StackTrace::StackTrace()','DB::InstrumentationManager::createTraceLogElement(DB::InstrumentationManager::InstrumentedPointInfo const&, XRayEntryType, std::__1::chrono::time_point>>) const','DB::InstrumentationManager::profile(XRayEntryType, DB::InstrumentationManager::InstrumentedPointInfo const&)','DB::InstrumentationManager::dispatchHandlerImpl(int, XRayEntryType)','DB::InstrumentationManager::dispatchHandler(int, XRayEntryType)','__xray_FunctionEntry','DB::QueryMetricLog::startQuery(std::__1::basic_string, std::__1::allocator> const&, std::__1::chrono::time_point>>, unsigned long)','DB::logQueryStart(std::__1::chrono::time_point>> const&, std::__1::shared_ptr const&, std::__1::basic_string, std::__1::allocator> const&, unsigned long, std::__1::shared_ptr const&, DB::QueryPipeline const&, DB::IInterpreter const*, bool, std::__1::basic_string, std::__1::allocator> const&, std::__1::basic_string, std::__1::allocator> const&, bool)','DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::__1::unique_ptr>&, std::__1::shared_ptr&, std::__1::shared_ptr, std::__1::function)','DB::executeQuery(std::__1::basic_string, std::__1::allocator> const&, std::__1::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum)','DB::TCPHandler::runImpl()','DB::TCPHandler::run()','Poco::Net::TCPServerConnection::start()','Poco::Net::TCPServerDispatcher::run()','Poco::PooledThread::run()','Poco::ThreadImpl::runnableEntry(void*)','start_thread','__clone3'] lines: ['./build/../src/Common/StackTrace.cpp:395','./src/Common/StackTrace.h:62','./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:738','./build/./src/Interpreters/InstrumentationManager.cpp:257','./build/./src/Interpreters/InstrumentationManager.cpp:225','/home/ubuntu/ClickHouse/ClickHouse/build/programs/clickhouse','./build/./src/Interpreters/QueryMetricLog.cpp:0','./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:667','./build/./src/Interpreters/executeQuery.cpp:0','./build/./src/Interpreters/executeQuery.cpp:0','./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:744','./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:583','./build/../base/poco/Net/src/TCPServerConnection.cpp:54','../contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:80','./build/../base/poco/Foundation/src/ThreadPool.cpp:219','../base/poco/Foundation/include/Poco/AutoPtr.h:77','/home/ubuntu/ClickHouse/ClickHouse/build/programs/clickhouse','/home/ubuntu/ClickHouse/ClickHouse/build/programs/clickhouse'] function_id: 231255 function_name: DB::QueryMetricLog::startQuery(std::__1::basic_string, std::__1::allocator> const&, std::__1::chrono::time_point>>, unsigned long) handler: profile entry_type: Exit duration_nanoseconds: 58435 ```

Converting to Chrome Event Trace Format

The profiling data can be converted to Chrome's Event Trace Format with the following query. Save the query to a `chrome_trace.sql` file: ```sql theme={null} WITH traces AS ( SELECT * FROM system.trace_log WHERE event_date >= today() AND trace_type = 'Instrumentation' AND handler = 'profile' ORDER BY event_time, entry_type ) SELECT format( '{{"traceEvents": [{}\n]}}', arrayStringConcat( groupArray( format( '\n{{"name": "{}", "cat": "clickhouse", "ph": "{}", "ts": {}, "pid": 1, "tid": {}, "args": {{"query_id": "{}", "cpu_id": {}, "stack": [{}]}}}},', function_name, if(entry_type = 0, 'B', 'E'), timestamp_ns/1000, toString(thread_id), query_id, cpu_id, arrayStringConcat(arrayMap((x, y) -> concat('"', x, ': ', y, '", '), lines, symbols)) ) ) ) ) FROM traces; ``` And executing it with ClickHouse Client to export it to a `trace.json` file that we can import either with [Perfetto](https://ui.perfetto.dev/) or [speedscope](https://www.speedscope.app/). ```bash theme={null} echo $(clickhouse client --query "$(cat chrome_trace.sql)") > trace.json ``` We can omit the stack part if we want a more compact but less informative trace.

See Also

* [SYSTEM INSTRUMENT](/reference/statements/system#instrument) — Add or remove instrumentation points. * [system.instrumentation](/reference/system-tables/instrumentation) — Inspect instrumented points. * [system.symbols](/reference/system-tables/symbols) — Inspect symbols to add instrumentation points.