CHECK_NRPE: Unclear error message during SSL mismatch

[tjyang]

This is a test case using nagios server and nrpe agent on same host.

• error message

[pi@nagios ~]$ check_nrpe -H localhost -n
CHECK_NRPE: Receive header underflow - only -1 bytes received (4 expected).
[pi@nagios ~]$ which check_nrpe
/usr/lib64/nagios/plugins/check_nrpe
[pi@nagios ~]$

[pi@nagios ~]$  check_nrpe -H localhost   -c check_logfiles_4A -a "/var/log/nagios/nagios.log" "nagios-log" ".*] Error:.*" ".*NOERROR.*"
CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected).
[pi@nagios ~]$

• Good check_nrpe results

[pi@nagios ~]$ check_nrpe -H localhost  -c check_cpu
OK - user: 0.74, nice: 0.50, sys: 1.48, iowait: 0.50, irq: 0.50, softirq: 0.50 idle: 99.27 | 'user'=0.74 ' <snipped>
[pi@nagios ~]$

[pi@nagios4 src]$ grep check_time   /etc/nrpe.d/check_time.cfg
command[check_time]=/usr/lib64/nagios/plugins/check_time.sh -H '$ARG1$' -w '$ARG2$' -c '$ARG3$'

[pi@nagios4 src]$ check_nrpe -H localhost   -c check_time -a localhost 1 2
OK - timedrift=0 seconds | timedrift=0s;1;2
[pi@nagios4 src]$

• OS and Nagios server info

[pi@nagios ~]$ rpm -qa |egrep 'nrpe-3.2.1|nagios-4.4.5';cat /etc/redhat-release;date
nagios-plugins-nrpe-3.2.1-8.el7.x86_64
nagios-4.4.5-1.el7.x86_64
nrpe-3.2.1-8.el7.x86_64
CentOS Linux release 7.7.1908 (Core)
Fri Nov  1 06:49:26 CDT 2019
[pi@nagios ~]$

• nrpe info

[pi@nagios ~]$  /usr/lib64/nagios/plugins/check_nrpe -H localhost
NRPE v3.2.1
[pi@nagios ~]$ 
[pi@nagios ~]$ egrep '^dont_blame_nrpe'  /etc/nagios/nrpe.cfg
dont_blame_nrpe=1
[pi@nagios ~]$
[pi@nagios ~]$ egrep 'logfile'  /etc/nrpe.d/check_logfile.cfg
command[check_logfiles_4A]=/usr/lib64/nagios/plugins/contrib/check_logfiles  --logfile $ARG1$ --tag $ARG2$ --warningpattern $ARG3$  --criticalpattern  $ARG4$
[pi@nagios ~]$
[pi@nagios ~]$ egrep 'cpu'  /etc/nrpe.d/check_cpu.cfg
command[check_cpu]=/usr/lib64/nagios/plugins/check_cpu.sh
[pi@nagios ~]$

[tjyang]

Hi

Can you confirm following issue as bug ?

• reqular expression patter within double qoutes doesn't work.

[pi@nagios4 ~]$ check_nrpe -H localhost   -c check_logfiles_4A -a "/var/log/nagios/nagios.log" "nagios-log" ".*[^]] Error:.*" ".*NOERROR.*"
CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected).
[pi@nagios4 ~]$ 

• work arounds that works.

[pi@nagios4 ~]$ check_nrpe -H localhost   -c check_logfiles_4A -a "/var/log/nagios/nagios.log" "nagios-log" .*[^]] Error:.*  .*NOERROR.*                 
OK - no errors or warnings|nagios-log_lines=0 nagios-log_warnings=0 nagios-log_criticals=0 nagios-log_unknowns=0
[pi@nagios4 ~]$ check_nrpe -H localhost   -c check_logfiles_2A_local -a "/var/log/nagios/nagios.log" "nagios-log"
OK - no errors or warnings|nagios-log_lines=0 nagios-log_warnings=0 nagios-log_criticals=0 nagios-log_unknowns=0
[pi@nagios4 ~]$ grep local /etc/nrpe.d/check_logfile_local.cfg
command[check_logfiles_2A_local]=/usr/lib64/nagios/plugins/contrib/check_logfiles  --logfile $ARG1$ --tag $ARG2$ --warningpattern  .*[^]] Error:.*  --criticalpattern  .*NOERROR.*
[pi@nagios4 ~]$ date
Thu Nov  7 16:47:21 CST 2019
[pi@nagios4 ~]$

[sawolf]

Hi @tjyang - thanks for reporting this. I'm not too familiar with NRPE, so hopefully @hedenface will be able to tell you whether or not that's a bug (though I agree that it looks like a system-level error).

One thing I did notice, though - removing the quotes should have the same effect as running the command like

check_nrpe -H localhost   -c check_logfiles_4A -a "/var/log/nagios/nagios.log" "nagios-log" ".*[^]]" "Error:.*"  ".*NOERROR.*"

that is, the whitespace would separate different shell arguments. Also, since you're invoking these directly on the shell, you may be getting issues related to globbing expansion. I would try using single quotes in this situation, i.e.

check_nrpe -H localhost   -c check_logfiles_4A -a "/var/log/nagios/nagios.log" "nagios-log" '.*[^]] Error:.*'  '.*NOERROR.*'

[sawolf]

I just talked to Bryan - he recommends taking a look at your SSL settings, both on the check_nrpe and NRPE sides. Your other commands aren't using the -n option, so that may be the source of your problems. He also says to check the version numbers - I think it's unlikely to be an issue since you're running it against localhost, but may be worth a shot.

[hedenface]

Why are you using -n in the first command you list but no others? Let's see ./check_nrpe -H localhost (without the -n).

[tjyang]

@hedenface

"-n, --no-ssl Do no use SSL"

• Why are you using -n in the first command you list but no others?

I was hoping to see the output from with/without -n should be the same.

[pi@nagios4 ~]$ check_nrpe -H localhost ; check_nrpe -H localhost -n
NRPE v3.2.1
CHECK_NRPE: Receive header underflow - only -1 bytes received (4 expected).
[pi@nagios4 ~]$

[sawolf]

If you want to use non-SSL connections, you need to make sure the NRPE daemon is also running with -n. The check_nrpe client currently doesn't do anything to determine whether there's an SSL mismatch, hence the errors. I do agree that the error messages could be clearer, though.

[tjyang]

Thanks @sawolf to put this issue as an RFE.

• "-n" added in nrpe.service to allow no ssl connection.

[pi@nagios4 ~]$ egrep "\-n"  /usr/lib/systemd/system/nrpe.service
ExecStart=/usr/sbin/nrpe -n -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT
[pi@nagios4 ~]$

[pi@nagios4 ~]$  check_nrpe -H localhost -n
NRPE v3.2.1
[pi@nagios4 ~]$

[tjyang]

Now back to the real goal I want to achieve, ie. to pass over Regular Expression pattern for check_logfile command.

• remove all nasty metachars blocking between two double quotes below.

[pi@nagios4 ~]$ grep nasty /etc/nagios/nrpe.cfg
nasty_metachars=""
[pi@nagios4 ~]$

• successful log after nrpe daemon restart.

[pi@nagios4 ~]$  check_nrpe -H localhost   -c check_logfiles_4A -a "/var/log/nagios/nagios.log" "nagios-log" ".*[^]] Error:.*"  ".*NOERROR.*"
OK - no errors or warnings|nagios-log_lines=0 nagios-log_warnings=0 nagios-log_criticals=0 nagios-log_unknowns=0
[pi@nagios4 ~]$

• suggestion: return error should related RE characters was blocked by setting in /etc/nagios/nrpe.cfg.