S3 Remote Backup Just Storing metadata

[DantasB]

Hey guys! I need some help on trying to run the create_remote command.

Basically my infrastructure is:

1. One clickhouse running in a production account.
2. One clickhouse running in a staging account.
3. One backup-bucket in a shared account. (Encripted by a KMS key hosted in encryption account)

The idea is:

1. Run the create_remote for some tables in the production account and store this backup in the shared account bucket.
2. Run the restore_remote for every table data in the backup located in the shared account bucket (i just want to transfer the data).

The point is, currently, when i run the create_remote when i look at my S3 bucket, there's just the metadata information, so, there's no "data" on it. And, consequently, when i run the restore_remote, the table remains with zero data.

This is the command that i'm running for create_remote:

docker run -u $(id -u clickhouse) --rm -it --network clickhouse-net -v "/var/lib/clickhouse:/var/lib/clickhouse" -e CLICKHOUSE_HOST="local_clickhouse_ip" -e S3_BUCKET="shared-bucket-name"  -e REMOTE_STORAGE="s3" -e S3_SSE="aws:kms" -e S3_SSE_KMS_KEY_ID="custom_managed_key_arn" -e S3_ACL="" -e BACKUPS_TO_KEEP_LOCAL="-1" altinity/clickhouse-backup:2.5.14 create_remote --tables "database.table_name,database.table_name_2"

This is the command that i'm running for restore_remote:

docker run -u $(id -u clickhouse) --rm -it --network clickhouse-net -v "/var/lib/clickhouse:/var/lib/clickhouse" -e CLICKHOUSE_HOST="local_clickhouse_ip" -e S3_BUCKET="shared-bucket-name"  -e REMOTE_STORAGE="s3" -e S3_SSE="aws:kms" -e S3_SSE_KMS_KEY_ID="custom_managed_key_arn" -e S3_ACL="" -e BACKUPS_TO_KEEP_LOCAL="-1" altinity/clickhouse-backup:2.5.14 restore_remote --data --tables "database.table_name,database.table_name_2" backup_name_on_s3

The clickhouse version is: 20.12.4.5
The clickhouse-backup version is: 2.5.14

My config is the default, except for the parameters used on docker run command.

What am i configuring wrong?

[Slach]

Everything looks correct, at the first look
could you add -e "LOG_LEVEL=debug"
and share full logs for create_remote and restore_remote execution?

[DantasB]

Oh, an important info: i just have one disk in this clickhouse instance. That's the default. (/var/lib/clickhouse)

[Slach]

tcp://clickhouse-docker-ip:9000
is clickhouse-docker-ip docker container running is the same physical host with clickhouse-backup?

is this container also have volume mapping /var/lib/clickhouse from host?

could you compare and share
ls -la /var/lib/clickhouse/data/database/table_namefrom host and insideclickhouse-server` container?

[DantasB]

I saw the volume bind and it's really stored in another place locally.

[{bind  /mnt/clickhouse/clickhouse-data /var/lib/clickhouse   true rprivate} {bind  /mnt/clickhouse/clickhouse-config /etc/clickhouse-server   true rprivate} {bind  /mnt/clickhouse/clickhouse-log /var/log/clickhouse-server   true rprivate}]

The /var/lib/clickhouse-data is the folder inside the ec2 instance and the /var/lib/clickhouse is in the container.

[Slach]

sorry still not clear
looks like you need run your clickhouse-backup container with -v /mnt/clickhouse/clickhouse-data:/var/lib/clickhouse instead of -v /var/lib/clickhouse/:/var/lib/clickhouse

ensure clickhouse-backup and clickhouse-server look to the same data

[DantasB]

Yes, exactly what you said, the mapping of the update command is wrong. I'll try that and send you any updates.

Thanks for helping!!