Besides the metrics from the data points from info, memory and the latency framework in the sections above, you may need to pull data from other sources when troubleshooting.
The Redis server will respond to the PING command when running properly:
$ redis-cli -h redis.example.com -p 6379 PING
PONGRedis Slow Log is a system to log queries that exceed a specific execution time which does not include I/O operations like client communication. It is enabled by default with two configuration parameters.
slowlog-log-slower-than 1000000This indicates if there is an execution time longer than the time in microseconds, in this case one second, it will be logged. The slow log can be disabled using a value of -1. It can also be set to log every command with a value of 0.
slowlog-max-len 128This sets the length of the slow log. When a new command is logged the oldest one is removed from the queue.
These values can also be changed at runtime using the CONFIG SET command.
You can view the current length of the slow log using the LEN subcommand:
redis.cloud:6379> slowlog len
(integer) 11Entries can be pulled off of the slow log using the GET subcommand.
redis.cloud:6379> slowlog get 2
1) 1) (integer) 10
2) (integer) 1616372606
3) (integer) 600406
4) 1) "debug"
2) "sleep"
3) ".6"
5) "172.17.0.1:60546"
6) ""
2) 1) (integer) 9
2) (integer) 1616372602
3) (integer) 600565
4) 1) "debug"
2) "sleep"
3) ".6"
5) "172.17.0.1:60546"
6) ""The slow log can be reset using the RESET subcommand.
redis.cloud:6379> slowlog reset
OK
redis.cloud:6379> slowlog len
(integer) 0There are a few options that can be passed to redis-cli that will trigger a keyspace analysis. They use the SCAN command so they should be safe to run without impacting operations. You can see in the output of all of them there is a throttling option if needed.
Key Stats (--keystats): This option will scan the dataset for keys based on memory size and length.
You can use the --keystats or --keystats-samples <n> options for this.
$ redis-cli --keystats
# Scanning the entire keyspace to find biggest keys and distribution information.
# Use -i 0.1 to sleep 0.1 sec per 100 SCAN commands (not usually needed).
# Use --cursor <n> to start the scan at the cursor <n> (usually after a Ctrl-C).
# Use --top <n> to display <n> top key sizes (default is 10).
# Ctrl-C to stop the scan.
100.00% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Keys sampled: 12
Keys size: 46.54K
--- Top 10 key sizes ---
1 3.90K hash "semantic-router:politics:64496d36ea3d10f4db5bb19578f8054c90b9a33a5ac7c7be7bc8f06f88bf990d"
2 3.89K hash "semantic-router:politics:672bd5232ae0cb6810db3a2c4590c393218d3b076313de7d0a96ccda010f31ef"
3 3.88K hash "semantic-router:pii:eb77e2b656d59aea1a3dc26fc7f7c6e577582d9e994ab2ccad1314ca4cea563f"
4 3.88K hash "semantic-router:aliens:8c6816d6115e047adca17426bea8a97977fe996cea14481bf9f549a44fcda74e"
5 3.88K hash "semantic-router:aliens:d54a060930a7de006034fb28ccc87bf9041a7c158760488e4b090fead22d1867"
6 3.88K hash "semantic-router:politics:c90afab0bd6f175342c30ada4d798657d4fc2cd3da6bb1404b20964d6ff9ca7d"
7 3.88K hash "semantic-router:politics:9cdcdcece82858b7627605ed81b68f115aa8cd7fbc2dac5b5f7e0f01b4c3573b"
8 3.88K hash "semantic-router:pii:9140b5958f790a6bdec0e1c4b342fe2620f28ef18dac4540080f157bc0a78e0b"
9 3.88K hash "semantic-router:pii:0299c9bbf9476c4f9667e2ceb6af59e7db8afce4803fb9566593b3263d317cae"
10 3.87K hash "semantic-router:pii:b1fdc8e3cd61a3a89055969a5c76f12d02e85a7f7be0976867f0df7135351355"
--- Top size per type ---
hash semantic-router:politics:64496d36ea3d10f4db5bb19578f8054c90b9a33a5ac7c7be7bc8f06f88bf990d is 3.90K
--- Top length and cardinality per type ---
hash semantic-router:politics:672bd5232ae0cb6810db3a2c4590c393218d3b076313de7d0a96ccda010f31ef has 3 fields
Key size Percentile Total keys
-------- ---------- -----------
3.86K 0.0000% 2
3.88K 50.0000% 10
3.89K 87.5000% 11
3.90K 100.0000% 12
Note: 0.01% size precision, Mean: 3.88K, StdDeviation: 11B
Key name length Percentile Total keys
--------------- ---------- -----------
89B 100.0000% 12
Total key length is 1.02K (86B avg)
Type Total keys Keys % Tot size Avg size Total length/card Avg ln/card
--------- ------------ ------- -------- -------- ------------------ -----------
hash 12 100.00% 46.54K 3.88K 36 fields 3.00Big Keys (--bigkeys): This option will scan the dataset for big keys and provide information about them.
Mem Keys (--memkeys): Similarly to big keys, mem keys will look for the biggest keys but also report on the average sizes.
Prior to --keystats (before Redis 7.4.0) you need to use a combination of --bigkeys and --memkeys to achieve the same thing.
Hot Keys (--hotkeys): The hot keys scan is only available when the maxmemory-policy is set to volatile-lfu or allkeys-lfu. If you need to identity hot keys you can add this argument to redis-cli.
$ redis-cli --hotkeys
# Scanning the entire keyspace to find hot keys as well as
# average sizes per key type. You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).
[00.00%] Hot key '"key:__rand_int__"' found so far with counter 37
-------- summary -------
Sampled 5 keys in the keyspace!
hot key found with counter: 37 keyname: "key:__rand_int__"Monitor: The MONITOR command allows you to see a stream of every command running against your Redis instance.
127.0.0.1:6379 > monitor
OK
1616541192.039933 [0 127.0.0.1:57070] "PING"
1616541276.052331 [0 127.0.0.1:57098] "set" "user:2398423hu" "KutMo"Since MONITOR streams back all commands, its use comes at a cost. It has been known to reduce performance by up to 50% so use with caution!
The Redis log file is the other important log you need to be aware of. It contains useful information for troubleshooting configuration and deployment errors. If you don't configure Redis logging, troubleshooting will be significantly harder.
Redis has four logging levels, which you can configure directly in redis.conf file.
Log Levels:
WARNINGNOTICEVERBOSEDEBUGRedis also supports sending the log files to a remote logging server through the use of syslog.
Remote logging is important to many security professionals. These remote logging servers are frequently used to monitor security events and manage incidents. These centralized log servers perform three common functions: ensure the integrity of your log files, ensure that logs are retained for a specific period of time, and to correlate logs against other system logs to discover potential attacks on your infrastructure.
Let's set up logging on our Redis deployment. First we'll open our redis.conf file:
$ sudo vi /etc/redis/redis.confThe redis.conf file has an entire section dedicated to logging.
First, find the logfile directive in the redis.conf file. This will allow you to define the logging directory. For this example lets use /var/log/redis/redis.log.
If you'd like to use a remote logging server, then you'll need to uncomment the lines syslog-enabled, syslog-ident and syslog-facility, and ensure that syslog-enabled is set to yes.
Next, we'll restart the Redis server.
You should see the log events indicating that Redis is starting.
$ sudo tail -f /var/log/redis/redis.logAnd next let's check that we are properly writing to syslog. You should see these same logs.
$ less /var/log/syslog | grep redisFinally, you’ll need to send your logs to your remote logging server to ensure your logs will be backed up to this server. To do this, you’ll also have to modify the rsyslog configuration. This configuration varies depending on your remote logging server provider.
