In real deployments, without atomic clocks, the time synchronized by NTP can drift, and servers in a distributed system can show a clock skew of hundreds of milliseconds. A simple way to test this in a Docker lab is to fake the clock_gettime function. Here is an example with a 2-node RF1 YugabyteDB cluster (PostgreSQL-compatible Distributed SQL database).
I create a yb network and start the first node, yb1 in the background:
docker network create yb
docker run -d --rm --network yb --hostname yb1 -p 7000:7000 yugabytedb/yugabyte yugabyted start --background=false --tserver_flags="TEST_docdb_log_write_batches=true"
I start a shell in a second node:
docker run -it --rm --network yb --hostname yb2 yugabytedb/yugabyte bash
In this container, I wait to be sure that yb1 is up and start yb2 that joins yb1
until postgres/bin/pg_isready -h yb1.yb ; do sleep 1 ; done
yugabyted start --join yb1.yb --tserver_flags="TEST_docdb_log_write_batches=true"
Here, running on the same host, both containers show the same Physical Time in http://localhost:7000/tablet-server-clocks
I install gcc and compile a fake_clock_gettime.so that overrides clock_gettime, calls the original one, and subtracts 499 milliseconds to its result:
cat > fake_clock_gettime.c <<'C'
#define _GNU_SOURCE
#include <stdlib.h>
#include <dlfcn.h>
int clock_gettime(clockid_t clk_id, struct timespec *tp)
{
static int skew_millisecond = 499;
static int (*origin_clock_gettime)();
static int ret;
// define the real clock_gettime and call it
if(!origin_clock_gettime) {
origin_clock_gettime = (int (*)()) dlsym(RTLD_NEXT, "clock_gettime");
}
ret=origin_clock_gettime(clk_id,tp);
// add clock skew and return
if (tp->tv_nsec >= skew_millisecond * 1000000 ) {
tp->tv_nsec -= skew_millisecond * 1000000 ;
} else {
tp->tv_sec -= 1;
tp->tv_nsec += 1000000000 - skew_millisecond * 1000000 ;
}
return(ret);
}
C
dnf install -y gcc
gcc -o fake_clock_gettime.so -fPIC -shared fake_clock_gettime.c -ldl
This library can be loaded with LD_PRELOAD, and I test it by calling date:
[root@yb2 yugabyte]# date +"%T:%N" ; LD_PRELOAD=$PWD/fake_clock_gettime.so date +"%T:%N" ; date +"%T:%N"
21:31:44:015385334
21:31:43:518894559
21:31:44:020271039
[root@yb2 yugabyte]# date +"%T:%N" ; LD_PRELOAD=$PWD/fake_clock_gettime.so date +"%T:%N" ; date +"%T:%N"
21:31:45:955772746
21:31:45:459189786
21:31:45:960576587
The date called with the library shows a lower time.
I restart YugabyteDB on yb2 with this hack:
yugabyted stop
LD_PRELOAD=$PWD/fake_clock_gettime.so yugabyted start
I can see the clock skew on the Physical Time and Hybrid Time:
I run some workload that involves tablets in both nodes to get some Lamport logical clock synchronization:
/home/yugabyte/postgres/bin/ysql_bench -i -h $(hostname) -s 10
With the messaging between the nodes, the Physical Time still shows a clock skew, but the Logical Time is synchronized:
If you are curious, here is more information about clock synchronisation in distributed databases: https://www.yugabyte.com/blog/evolving-clock-sync-for-distributed-databases/
Top comments (0)