Maintenance

Maintenance - Debugging Calls

[Ref: OpenBSD 5.1, Asterisk 1.8.11 (from ports) | Debugging failed Asterisk calls outside the local network | ]

Table of Contents

DNS

[Ref: resolv.conf(5)]

DNS, Name Resolution matters. Make sure you have configured your name resolution correctly.

We’ve encountered the SIP module loading very slowly without the correct DNS/Name Resolution configuration.

It is good practise to make sure that you have your DNS/Name resolution working efficiently. The general configurations to review include:

  • resolv.conf
  • hosts

Is Asterisk taking a long time to load your modules? What real implications are there for misconfigured DNS settings?

In our environment, we went from 5 minutes to load channel_sip.so to almost instantaneous. Good enough reason to check.

resolv.conf

[Ref: resolv.conf(5)]

Make sure that the ip addresses you’ve set up as your nameservers are

  • routable, can you connect to them
  • responding to DNS, are they actually DNS servers?

File extract: /etc/resolv.conf

1
2
3
4
search example.com
nameserver ip-address-1
nameserver ip-address-2
lookup file bind

hosts

[Ref: hosts(5)]

You can speed up DSN resolution of the host by making sure that it is correct in /etc/hosts.

File extract: /etc/hosts

1
2
3
127.0.0.1       localhost
::1             localhost
host-ip-address pbxhost.example.com pbxhost

The above ‘host-ip-address’ should obviously match the ip-address you’ve configured for your host on at least one of your network connections.

[Ref: hostname.if(5)]

File extract: /etc/hostname.nic1

1
inet host-ip-address host-ip-netmask host-ip-broadcast

Logs

The Console and logs are fundamental for to operational review maintenace, and monitoring the system behaviour, performance.

  • Logs are stored in: /var/log/asterisk
  • The console is accessed through: asterisk -r

In general operations, it is good to review the messages log

Log Files

For historical information, and to monitor error messages, keep a lookout on the log messages.

1
$ sudo tail -f /var/log/asterisk/message

You may have to look at the log configuration, in case someone has changed it from the default. If logging is disabled. After reviewing the configuration file (/etc/asterisk/logger.conf) remember to activate the changes by applying:

1
$ sudo asterisk -r -x "logger reload"

If you’re ’traffic’ is too noisy, then you can filter your log viewing in many different ways such as:

1
$ sudo cat /var/log/asterisk/messages | grep ERROR

Command-line

[Ref: AST CLI Connecting to the Asterisk CLI ]

From the manpage:

       -c     Provide  a control console on the calling terminal. The console is similar to the remote console provided
              by -r. Specifying this option implies -f and will cause asterisk to no longer fork  or  detach  from  the
              controlling terminal. Equivalent to console = yes in asterisk.conf.
       -r     Instead of running a new Asterisk process, attempt to connect to a running Asterisk process and provide a
              console interface for controlling it.
       -R     Much like -r. Instead of running a new Asterisk process, attempt to connect to a running Asterisk process
              and provide a console interface for controlling it. Additionally, if connection to the  Asterisk  process
              is lost, attempt to reconnect for as long as 30 seconds.
              
1
$ sudo asterisk -r -x "reload"
1
$ sudo asterisk -r -x "core show help" | grep -i "sip"
1
$ sudo asterisk -r -x "sip show peers | grep -v Unrecognised"

Console

[Ref: Debugging with Verbose, Changing the Verbose and Debug Levels]

We normally enter the Asterisk Console with debugging levels by using “-rvvv” where the number of ‘v’s indicates the verbosity or debug mode we wish the console to display.

1
$ sudo asterisk -rvvvv

Inside the console, we can change the settings directly using core set debug.

1
2
server*CLI> core set debug 4
Core debug was 0 and is now 4

Module Options

The console provides a command-line prompt allowing commands executed against the running asterisk session.

Most modules have a command-line help screen that can be found using core show help command. For example, to find out what CLI options are available from the dialplan module,

1
server*CLI> core show help dialplan

A fundamental console tool is the ‘console’

1
server*CLI> core show help console
console answer                 -- Answer an incoming console call
console dial                   -- Dial an extension from the console
console flash                  -- Send a flash to the connected party
console hangup                 -- Hangup a call on the console
console {mute|unmute}          -- Disable/Enable mic input
console list available         -- List available devices
console list devices           -- List configured devices
console send text              -- Send text to a connected party
console {set|show} autoanswer [on|off] -- Turn autoanswer on or off
console {set|show} active      -- View or Set the active console device
1
voip2*CLI> core show help console dial
Usage: console dial [extension[@context]]
       Dials a given extension (and context if specified)
1
console dial 5500@phones
 -- Executing [5500@stdexten:1] Dial("Console/default", "SIP/5500,60") in new stack
 -- Called SIP/5500
1
voip2*CLI> core show help channel originate
  There are two ways to use this command. A call can be originated between a
channel and a specific application, or between a channel and an extension in
the dialplan. This is similar to call files or the manager originate action.
Calls originated with this command are given a timeout of 30 seconds.

Usage1: channel originate  application  [appdata]
  This will originate a call between the specified channel tech/data and the
given application. Arguments to the application are optional. If the given
arguments to the application include spaces, all of the arguments to the
application need to be placed in quotation marks.

Usage2: channel originate  extension [exten@][context]
  This will originate a call between the specified channel tech/data and the
given extension. If no context is specified, the 'default' context will be
used. If no extension is given, the 's' extension will be used.

With channel originate, we can connect a SIP extension to an application (such as a recording, to verify the SIP channel, and the recording)

1
voip2*CLI> channel originate SIP/5500 application playback tt-weasels
    -- Called 5500
    -- SIP/5500-00000015 is ringing
    -- SIP/5500-00000015 answered
    --  Playing 'tt-weasels.slin' (language 'en')

Validate you can call between extensions

1
channel originate SIP/${MYEXTENSION} ${ANOTHEREXTENSION}@phones-context
    -- Called ${ANOTHEREXTENSION}
    -- SIP/${ANOTHEREXTENSION}-00000015 is ringing
    -- SIP/${ANOTHEREXTENSION}-00000015 answered SIP/${MYEXTENSION}
1
channel originate SIP/${MYLANDLINE}@${MYTRUNK} application playback tt-weasels

Command-line Filtering

You may need to process the results of an Asterisk CLI command, from the shell prompt.

1
$ sudo asterisk -rx 'core show help dialplan'

The above will let you make quick visits to the results, without having to stay within Asterisk.

Watch number of active calls

[Ref: Watch active calls on Asterisk PBX ]

1
$ gnuwatch -n 1 "sudo asterisk -vvvvvrx 'core show channels' | egrep \"(call|channel)\""

Watch active channels

1
$ gnuwatch -n 1 "sudo asterisk -vvvvvrx 'core show channels verbose'"

Dialplan

The dialplan is the foundation, bread and butter, of our Asterisk installation. It can seem a mystery why one number dials through and another totally disappears.

  • /var/log/asterisk/messages
  • core show help dialplan
  • dialplan reload
  • dialplan show

Many messages from the dialplan (errors/info) are logged so using the log files will let you see messages no longer available on the console.

1
$ sudo tail -f /var/log/asterisk/messages

help

As mentioned above, the help information provides a quick summary of available dialplan commands

1
server*CLI> core show help dialplan

for a list of dialplan commands

reload

One of the things we have to remember, whenever you update your dialplan (e.g. /etc/asterisk/extensions.conf or any #include ) we need to load the new configuration into the running application.

1
server*CLI> dialplan reload

for a list of dialplan commands

show

We use dialplan show with the following options

1
server*CLI> dialplan show <exten>@<context>

Use the above to drill down on what the dialplan will do, as opposed to how you think the dialplan will do things.

When you don’t have the explicit @ then you can use just the “@” without a context and Asterisk will try to find the context your number would be appropriately handled.

For example:

I want to see whether an extension is picked up by the system.

1
server*CLI> dialplan show 6521@
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
[ Context 'trunk_dial' created by 'pbx_config' ]
'_X.' => .
.

[ Content 'localexten' created by 'pbx_config' ]
'_X.' => .
.

[ Content 'friendpbx_dial' created by 'pbx_config' ]
'_X.' => .
.

[ Content 'internal' created by 'pbx_config' ]
'_65XX' => .
.

Whoa, that’s surprising.

Basically, the number can be picked up by a number of contexts, depending on who comes first and how each of the above context’s are reached.

I ignore the 1st three contexts, because they are contexts called by extensions (i.e. a call will not fall through to those extensions.)

So, the only valid context from the above list is ‘internal’. SUCCESS.

I want to see if I can call home? 0211111111 (yes, that is not a valid number)

1
server*CLI> dialplan show 0211111111@
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
[ Context 'trunk_dial' created by 'pbx_config' ]
'_X.' => .
.

[ Content 'localexten' created by 'pbx_config' ]
'_X.' => .
.

[ Content 'friendpbx_dial' created by 'pbx_config' ]
'_X.' => .
.

Oops, looks like that number can never be reached? Why?

If we take a look at the message file, we should see something like the below:

File extract: /var/log/asterisk/messages

[date] NOTICE[nnnnn] chan_sip.c: Call from '6521' (my-ip-address) to extension '0211111111' rejected because extension not found in context 'authorised-au-domestic'

Where ‘authorised-au-domestic’ is the special context for phones/extensions in our environment that are allowed to make calls to the domestic phone system.

Why can’t I call out? 02-1111-1111 is my home number!!!! I call it all the time!!!

1
2
3
server*CLI> dialplan show 0211111111@authorised-au-domestic
There is no existence of 0211111111@authorised-au-domestic
Command 'dialplan show 0211111111@authorised-au-domestic' failed

I wrack my brain, to realise that our defined dialplans require a ‘0’ as a prefix for calling external numbers (211111111) is not a valid numbering sequence in Australia. Some people choose ‘9’ as the outgoing call ‘prefix,’ the important thing is that the above will lead you to the failure, we still have to think it through.

1
server*CLI> dialplan show 00211111111@
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
[ Context 'trunk_dial' created by 'pbx_config' ]
'_X.' => .
.

[ Content 'localexten' created by 'pbx_config' ]
'_X.' => .
.

[ Content 'friendpbx_dial' created by 'pbx_config' ]
'_X.' => .
.

[ Content 'au-domestic' created by 'pbx_config' ]
'_00[235678]XXXXXXXX' => 1. Goto(trunk_dial,${EXTEN},1)    [pbx_config]

[ Included Content 'au-domestic' created by 'pbx_config' ]
'_00[235678]XXXXXXXX' => 1. Goto(trunk_dial,${EXTEN},1)    [pbx_config]

Ahhhh, so I can call home (with the outgoing call ‘prefix’ which is ‘0’) but I need to somehow be part of the ‘au-domestic’ context, which is included by at least one other context.

Client Configuration

From our sip.conf configuration for my extension ‘6521’, i find out that calls I make are dealt with in the ‘authorised-au-domestic’ context

File extract:

1
2
3
4
5
6
7
8
9
[basic-options](!)
type=friend
context=phones
.
[office-phones](!,basic-options)
context = authorised-au-domestic

[6521](office-phones)
mailbox = 6521

The results of the ‘dialplan show’ indicate that I can make the call, so somewhere in that context I do pickup the explicit context ‘au-domestic’

extensions.conf

Within the dialplan itself (for example /etc/asterisk/extensions.conf), you can add messages to help you view the progress through entries such as.

1
  same=> n,NoOP(Message ${DIALSTATUS}) 

and related output of Global Variables to provide diagnostic information on dial progression.

Queues

[Ref: Automatic Call Distribution (Queues) ]

  • load
  • help
  • reload
  • show

load

Once you’ve configured your queues in /etc/asterisk/queues.conf we need to load it into the running application

1
server*CLI> module reload app_queue.so

help

1
server*CLI> core show help queue

reload

1
server*CLI> queue reload

show

1
server*CLI> queue show

SIP

SIP devices, connections

  • help
  • reload
  • show

help

1
server*CLI> core show help sip

reload

Whenever we make changes to our SIP configuration, remember (as with other configurations) to ensure we load the configuration into the running application.

1
server*CLI> sip reload

show

There are a lof ‘show’ alternatives, but the two we use most often are:

  • sip show peers
  • sip show peer
1
server*CLI> sip show peers

With ‘show peers’ we quickly determine whether the device is connected, and the IP Address which is helpful for us to be able to log onto the devices web management console for further diagnostics.

The peer ‘status’ column is also quite helpful for our remote clients/branches giving us an indicator of the latency/quality of the link between the PBXs.

1
server*CLI> sip show peer <extension>

Provides us with further details about the particular peer/device and sometimes isolate whether with got a duplicate connection on an extension.

User VLANs

Remember that SIP is a cleartext communications protocol, so if you’re passing the username and password unencrypted, then it is not as difficult for someone between your client and Asterisk box to find your username/password and possibly intercept the audio, recording it for playback.

One mitigating tool, is to use a separate VLAN for voice traffic, as opposed to Data traffic.

Packet Dumps

[Ref: Using tcpdump for SIP diagnostic]

Sometimes, we really need to dig into the network packets to diagnose our PBX performance/behaviour.

The first part of diagnosis is the SIP (Session Initiation Protocol) exchange between the two devices (whether our PBX to a [soft]phone or another PBX.) Interpreting the SIP exchange can help determine SIP configuration errors.

OpenBSD’s tcpdump(8) does not have a SIP filter or ‘-T type’ but SIP traffic (over port 5060) is cleartext and ASCII output is a good start for diagnostics. OpenBSD 5.2’s tcpdump supports ASCII output, with the -A option.

-A  Print each packet in ASCII.  If the -e option is also
	specified, the link-level header will be included.  The smaller
	of the entire packet or snaplen bytes will be printed.

OpenBSD 5.1 (and earlier?) did not support the “-A” option, but you can use the “-X” option to give you the similar results.

1
$ sudo tcpdump -s 1500 -Xnqti em0 port 5060
-X  Print each packet in hex and ASCII.  If the -e option is also
	specified, the link-level header will be included.  The smaller
    of the entire packet or snaplen bytes will be printed.

As you struggle with understanding the SIP packet breakdown, and the hex dump output, you may find it easier to use Wireshark which has a nice structure display of the SIP packet output.

Use something similar to the below to output the SIP traffic to a file for offline review with Wireshark.

1
$ sudo tcpdump -s 1500 -w siptraffic.pcap -ni em0 port 5060

Your packet dumps can be modified to capture all traffic, and then you can also analyse the RTP traffic, which OpenBSD’s tcpdump(8) can help you interpret, or you can use Wireshark to replay the audio for further assessment.