ClickHouse INCORRECT_DATA Error on Fresh Install with JSON Column Type

[ihsanovali]

A fresh installation fails to ingest data into ClickHouse, resulting in a DB::Exception: Cannot parse JSON object here error. The issue occurs when the props column in the events table is set to the JSON data type. This leads to a type mismatch that halts the data ingestion process. The problem has been reproduced on a fresh deployment using docker-compose.

Reproduction

Environment:

• Checkout tag v1.5.1 (also tried on latest commit 2c54b0e)
• docker compose up -d --build
• ClickHouse Docker Image: clickhouse/clickhouse-server:25.4.2
• Docker Compose configuration with enable_json_type: 1

Steps to Reproduce:

• Start the application services using the provided docker-compose.yml.
• Ensure the ClickHouse events table is created with the props column as JSON.
• Trigger an event by making the following API call.

http://localhost:4200/api/v1/analytics/api/track
POST /api/v1/analytics/api/track HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 327
Content-Type: application/json
Host: localhost:4200
Origin: http://localhost:4200
Referer: http://localhost:4200/home
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36
sec-ch-ua: "Chromium";v="139", "Not;A=Brand";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Linux"

Request payload:

{
  "site_id": "1",
  "hostname": "localhost",
  "pathname": "/home",
  "querystring": "",
  "screenWidth": 1920,
  "screenHeight": 1080,
  "language": "en-US",
  "page_title": "My Page Title",
  "referrer": "",
  "user_id": "admin",
  "api_key": "rb_xxxxxxxxxxxxxxxxxxxxxxxx",
  "type": "custom_event",
  "event_name": "testEvent",
  "properties": "{\"testing\":\"123\"}"
}

The application responds with a 200 OK success message, but the data fails to ingest into ClickHouse.
Response 200: {"success":true}

Clickhouse Error Message

2025.08.20 07:14:16.746834 [ 65959 ] {d490f144-22bd-411a-9643-e2edeae33105} <Error> executeQuery: Code: 117. DB::Exception: Cannot parse JSON object here: {"testing":"123"}: (while reading the value of key props): (at row 1)
: While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.4.2.31 (official build)) (from 172.18.0.5:59138) (in query: INSERT INTO events FORMAT JSONEachRow ), Stack trace (when copying this message, always include the lines below):

This solves the problem but it breaks functionality of props:

ALTER TABLE events MODIFY COLUMN props String;

[goldflag]

I tried to repro this but i couldn't. I will keep on watching this issue to see if it pops up again though

[ihsanovali]

Thanks for the quick response!

I tracked this down, the issue wasn’t with ClickHouse itself but with the VM CPU configuration. On Ubuntu 24.04.3 LTS (GNU/Linux 6.8.0-78-generic x86_64) running inside Proxmox, I had the CPU type set to x86-64-v2-AES. That caused JSON parsing to fail in ClickHouse when inserting into a JSON column.

Even a simple JSON extract command would break inside the container:

docker exec clickhouse clickhouse-client -q "SELECT JSONExtractString('{\"testing\":\"123\"}', 'testing')"

Error:

Code: 173. DB::Exception: Received from localhost:9000. DB::Exception: Couldn't allocate 17 bytes when parsing JSON: In scope SELECT JSONExtractString('{"testing":"123"}', 'testing') AS t. (CANNOT_ALLOCATE_MEMORY)

I then tried running the same ClickHouse version on another server, and it worked fine.

Switching the VM’s CPU type in Proxmox from x86-64-v2-AES to host resolved the issue immediately. After that, ingestion into the props JSON column and the JSONExtractString function both worked as expected.

This looks like a memory allocation bug triggered by the CPU type emulation in Proxmox/KVM. Worth noting for others who might run into it on Ubuntu 24.04 under Proxmox.

For a thorough exploration of the underlying cause: ClickHouse/ClickHouse#66045
It appears that this issue may be resolved completely by itself in later versions of ClickHouse. Alternatively, we may need to add an additional configuration parameter.

[iammattmartin]

We were similarly caught out by this and were using x86-64-v2, setting to host also fixed the issue.

It broke tracking for us (#560).

[FoxxMD]

I'm having the same issue with json parsing

{"level":50,"time":1756216448655,"pid":1,"hostname":"ab8b8e4dd8c8","name":"rybbit","service":"pageview-queue","err":{"type":"ClickHouseError","message":"Cannot parse JSON object here: {\"helpful\":\"true\"}: (while reading the value of key props): (at row 1)\n: While executing ParallelParsingBlockInputFormat. ","stack":"Error: Cannot parse JSON object here: {\"helpful\":\"true\"}: (while reading the value of key props): (at row 1)\n: While executing ParallelParsingBlockInputFormat. \n    at parseError (/app/node_modules/@clickhouse/client-common/dist/error/parse_error.js:35:16)\n    at ClientRequest.onResponse (/app/node_modules/@clickhouse/client/dist/connection/node_base_connection.js:409:63)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","code":"117"},"msg":"Error processing pageview queue"}

clickhouse error

2025.08.26 13:54:08.654743 [ 1023327 ] {0db79961-7847-4478-8faa-c801dbc8fd10} <Error> DynamicQueryHandler: Code: 117. DB::Exception: Cannot parse JSON object here: {"helpful":"true"}: (while reading the value of key props): (at row 1)
: While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA), Stack trace (when copying this message, always include the lines below):

I'm also running rybbit on proxmox with CPU type x86-64-v3. I haven't tried switching to host yet because...i shouldn't have to update my entire VM just to parse json in one container.

@ihsanovali this is a truly wild find.

EDIT:

Cannot get the allow_simdjson setting to take affect. I'm having the same issue the OP has in the clickhouse issue where setting

<clickhouse>
<profiles>
        <default>
            <allow_simdjson>0</allow_simdjson>
        </default>
    </profiles>
</clickhouse>

does not actually change the setting during runtime.

[FoxxMD]

To set allow_simdjson=0, which fixes this issue, modify your rybbit docker-compose.yaml with the following:

Add an additional config in the top-level configs:

  clickhouse_user:
    content: |
      <clickhouse>
        <profiles>
          <default>
            <allow_simdjson>0</allow_simdjson>
          </default>
        </profiles>
      </clickhouse>

Add this to the clickhouse service configs

- source: clickhouse_user
  target: /etc/clickhouse-server/users.d/default.xml