> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-fix-nav-issues.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> Queries data to/from a remote HTTP/HTTPS server. This engine is similar to the File engine.

# URL table engine

Queries data to/from a remote HTTP/HTTPS server. This engine is similar to the [File](/reference/engines/table-engines/special/file) engine.

Syntax: `URL(URL [,Format] [,CompressionMethod])`

* The `URL` parameter must conform to the structure of a Uniform Resource Locator. The specified URL must point to a server that uses HTTP or HTTPS. This does not require any additional headers for getting a response from the server.

* The `Format` must be one that ClickHouse can use in `SELECT` queries and, if necessary, in `INSERTs`. For the full list of supported formats, see [Formats](/reference/formats/index#formats-overview).

  If this argument is not specified, ClickHouse detects the format automatically from the suffix of the `URL` parameter. If the suffix of `URL` parameter does not match any supported formats, it fails to create table. For example, for engine expression `URL('http://localhost/test.json')`, `JSON` format is applied.

* `CompressionMethod` indicates that whether the HTTP body should be compressed. If the compression is enabled, the HTTP packets sent by the URL engine contain 'Content-Encoding' header to indicate which compression method is used.

To enable compression, please first make sure the remote HTTP endpoint indicated by the `URL` parameter supports corresponding compression algorithm.

The supported `CompressionMethod` should be one of following:

* gzip or gz
* deflate
* brotli or br
* lzma or xz
* zstd or zst
* lz4
* bz2
* snappy
* none
* auto

If `CompressionMethod` is not specified, it defaults to `auto`. This means ClickHouse detects compression method from the suffix of `URL` parameter automatically. If the suffix matches any of compression method listed above, corresponding compression is applied or there won't be any compression enabled.

For example, for engine expression `URL('http://localhost/test.gzip')`, `gzip` compression method is applied, but for `URL('http://localhost/test.fr')`, no compression is enabled because the suffix `fr` does not match any compression methods above.

<h2 id="using-the-engine-in-the-clickhouse-server">
  Usage
</h2>

`INSERT` and `SELECT` queries are transformed to `POST` and `GET` requests,
respectively. For processing `POST` requests, the remote server must support
[Chunked transfer encoding](https://en.wikipedia.org/wiki/Chunked_transfer_encoding).

You can limit the maximum number of HTTP GET redirect hops using the [max\_http\_get\_redirects](/reference/settings/session-settings#max_http_get_redirects) setting.

<h2 id="example">
  Example
</h2>

**1.** Create a `url_engine_table` table on the server :

```sql theme={null}
CREATE TABLE url_engine_table (word String, value UInt64)
ENGINE=URL('http://127.0.0.1:12345/', CSV)
```

**2.** Create a basic HTTP server using the standard Python 3 tools and
start it:

```python3 theme={null}
from http.server import BaseHTTPRequestHandler, HTTPServer

class CSVHTTPServer(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/csv')
        self.end_headers()

        self.wfile.write(bytes('Hello,1\nWorld,2\n', "utf-8"))

if __name__ == "__main__":
    server_address = ('127.0.0.1', 12345)
    HTTPServer(server_address, CSVHTTPServer).serve_forever()
```

```bash theme={null}
$ python3 server.py
```

**3.** Request data:

```sql theme={null}
SELECT * FROM url_engine_table
```

```text theme={null}
┌─word──┬─value─┐
│ Hello │     1 │
│ World │     2 │
└───────┴───────┘
```

<h2 id="details-of-implementation">
  Details of Implementation
</h2>

* Reads and writes can be parallel
* Not supported:
  * `ALTER` and `SELECT...SAMPLE` operations.
  * Indexes.
  * Replication.

<h2 id="virtual-columns">
  Virtual columns
</h2>

* `_path` — Path to the `URL`. Type: `LowCardinality(String)`.
* `_file` — Resource name of the `URL`. Type: `LowCardinality(String)`.
* `_size` — Size of the resource in bytes. Type: `Nullable(UInt64)`. If the size is unknown, the value is `NULL`.
* `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`.
* `_headers` - HTTP response headers. Type: `Map(LowCardinality(String), LowCardinality(String))`.

<h2 id="resolving-relative-urls">
  Resolving relative URLs
</h2>

The [url\_base](/reference/settings/session-settings#url_base) setting allows using a relative URL in the `URL` engine. When `url_base` is set, the URL passed to the engine is resolved against it per [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986). For a full description of the resolution rules, see the [url table function docs](/reference/functions/table-functions/url#resolving-relative-urls).

**Example**

```sql theme={null}
SET url_base = 'http://127.0.0.1:12345/';
CREATE TABLE url_engine_table (word String, value UInt64) ENGINE = URL('hello.csv', CSV);
SELECT * FROM url_engine_table;
```

<h2 id="storage-settings">
  Storage settings
</h2>

* [engine\_url\_skip\_empty\_files](/reference/settings/session-settings#engine_url_skip_empty_files) - allows to skip empty files while reading. Disabled by default.
* [enable\_url\_encoding](/reference/settings/session-settings#enable_url_encoding) - allows to enable/disable decoding/encoding path in uri. Enabled by default.
* [url\_base](/reference/settings/session-settings#url_base) - base URL for resolving relative URLs passed to the engine.