Tunnel interface example:
!
interface Tunnel1
ip address 172.16.1.1 255.255.255.0 !! Select a private IP subnet for the tunnels
!
tunnel protection ipsec profile PROFSEC !! encrypts the traffic passing through this tunnel using ipsec
tunnel source GigabitEthernet0/0 !! source of the tunnel is the WAN interface
tunnel mode gre multipoint !! designates the tunnel as a mGRE tunnel
tunnel key 100000 !! pick the right tunnel (by key ID) before it can decapsulate the packet and look at the IP address.
!
ip nhrp authentication nhrp1234 !! (optional) authentication used for updates between the routers
ip nhrp network-id 1 !! (mandatory) network identification that has to be the same on all the routers
!
ip nhrp map multicast dynamic !! Enables multicast across the tunnel, hub replicates multicast packets to all spokes registered via NHRP
! Add spoke to multicast list upon registration
!
!
!
ip mtu 1400 !! Reduce the MTU to allow extra overhead from mGRE and IPSEC
load-interval 30 !! IOS calculates statistics (including load) over this interval (default 5 min)
!
!! Keepalives must be missed before the tunnel is shut down.
keepalive 5 10 !! [seconds [retries]] - Keepalives are sent every 5 seconds and 10 retries. (default is 10 3)
!
bandwidth 1000 !! default is 8 kbps, it may not have enough Bw when the number of spoke increases and you will see session flapping.
delay 1000 !! default is 50000 microsec
!
no ip split-horizon eigrp 1 !! disable split horizon for EIGRP
no ip next-hop-self eigrp 1 !! routing update with the next hop unmodified to the spokes
!
!
Tunnel Bandwidth
Cisco IOS/NX-OS/etc. software does not configure the bandwidth for a virtual tunnel interface based on the physical interface to which it is assigned; instead, it applies a default "bandwidth" statement to the interface that depends on model of hardware and the version of software it is running (on many devices the default "BW" for a tunnel is 8kbps) !
IOSv L3:
HUB# sh int Tunnel1 | inc BW|bandwidth
MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
HUB#
IOS 7200 Software , Version 15.1(4)M2
C_C7200#sh int tunnel 11 | inc BW|band
MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
C_C7200#
Other IOS's:
dyn5#sh int tunnel 0
Tunnel0 is up, line protocol is up
Hardware is Tunnel
Internet address is 10.10.10.5/24
MTU 1416 bytes, BW 9 Kbit, DLY 500000 usec,
int tunnel 1
bandwidth <bandwidth> ! Bandwidth in kbps
bandwidth 1000 ! Bandwidth is set to 1000 kbps = 1 mbps
tunnel bandwidth {receive | transmit} <bandwidth> ! specify the capacity of the satellite link.
delay <1-16777215> Throughput delay (tens of microseconds)
The show interface commands displays delay in microsecond units.
The delay interface command specifies the delay metric, in 10 microsecond units.
EIGRP calculates its metric from the minimum bandwidth in Kbps for all links in the path,
and the cumulative delay in microseconds for all links in the path.
Sources:
- https://www.networkstraining.com/cisco-dmvpn-configuration-example/
- https://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/41940-dmvpn.html
- http://blog.ine.com/2008/08/02/dmvpn-explained/
- https://learningnetwork.cisco.com/blogs/vip-perspectives/2017/02/15/dmvpn-the-phases-in-depth
- https://www.cisco.com/c/dam/en/us/products/collateral/security/dynamic-multipoint-vpn-dmvpn/dmvpn_design_guide.pdf
DMVPN uses the following group of networking technologies:
1) Multipoint GRE (mGRE)
2) Next-Hop Resolution Protocol (NHRP)
4) Dynamic Routing Protocol (EIGRP, RIP, OSPF, BGP) or static routing
3) Dynamic IPsec encryption
5) Cisco Express Forwarding (CEF)
Two major phases of DMVPN evolution:
1) Phase 1 – Only Hub to Spoke tunnel deployment (mGRE hub, p2p GRE spokes)
This means GRE tunnels are only built between the hub and the spokes. Traffic destined to networks behind spokes is forced to first traverse the hub.
2) Phase 2 – Hub-to-Spoke and Spoke-to-Spoke tunnels (mGRE everywhere)
Phase 2 improved on Phase 1 by allowing spokes to build a spoke-to-spoke tunnel on demand with these restrictions:
- Spokes must use multipoint GRE tunnels
- The spokes must receive specific routes for all remote spoke subnets
- The next hop of the entry in the routing table must list the remote spoke as the next hop
This is achieved by removing the static tunnel destination command and replacing it with the tunnel mode gre multipoint command.
Phase 2 with EIGRP, split horizon must be disabled on the tunnel interface using the no ip split-horizon eigrp [asn] command.
3) Phase 3 – "Scalable Infrastructure"
Though DMVPN Phase 2 deployment provided direct spoke-to-spoke tunnels, one of the limitations is maintaining full routing tables on the spokes. Each route for remote spoke networks needs to be a specific route with the next hop pointing to the remote spoke’s tunnel address. This prevents the hub from being able to send down a summarized route to the spokes for a more concise routing table.
Phase 3 overcomes this restriction using NHRP traffic indication messages from the hub to signal to the spokes that a better path exists to reach the target network. This functionality is enabled by configuring ip nhrp redirect on the hub and ip nhrp shortcut on the spokes. The redirect command tells the hub to send the NHRP traffic indication message while the shortcut command tells the spokes to accept the redirect and install the shortcut route.
The protocol header for an mGRE packet is four bytes larger than a p2p GRE packet. The additional four bytes constitute a tunnel key value, which is used to differentiate between mGRE interfaces in the same router. Without a tunnel key, a router can support only one mGRE interface, corresponding to one IP network. Tunnel keys enable a branch router to have a different mGRE interface corresponding to each DMVPN cloud in the network topology. A headend router can also be configured with two mGRE interfaces pointing to each DMVPN cloud for high availability and redundancy. Cisco IOS Software Releases 12.3(13)T, 12.3(11)T3, or later support the configuration of multiple mGRE interfaces on a single router without tunnel keys. In this case, each mGRE interface must reference a unique IP address as its tunnel source.
Tunnel Protection Mode
Tunnel protection is used to secure (encrypt) the data transfer inside the GRE tunnel by applying an IPsec profile to the mGRE tunnel interface. Crypto maps are unnecessary in IOS Release 12.2(13)T or later.
There is no need to specify the IPsec peer address or the ACL to match the packets that are to be encrypted. If a packet is routed through the tunnel, it is encrypted.
If more than one mGRE tunnel interface or an mGRE and p2pGRE tunnel interface is configured on a router, and they use the same tunnel source address, the shared keyword must be configured on the tunnel protection command.
NHRP Configuration
- Each router in topology acts as either NHC (Next-Hop Client) or NHS (Next-Hop Server).
- NHRP provides a mapping between the inside (VPN) and outside (NBMA) address of a tunnel endpoint.
- One of the functions of NHC is to register with NHS its IP address mapped to NBMA Layer 2 address (e.g. ATM NSAP address). To make registration possible, you configure each NHC with the IP address of at least one NHS. In turn, NHS acts as a database agent, storing all registered mappings, and replying to NHC queries. If NHS does not have a requested entry in its database, it can forward packet to another NHS to see if it has the requested association.
- A router may act as a Next-Hop server and client at the same time. a router may act as a Next-Hop server and client at the same time.
- NHRP entry has countdown expire timer, initialized from the registration hold-time. Every 60 seconds global NHRP process runs on a router and checks the expire timer on all NHRP entries. If the expire timer for an NHRP entry is greater than 120 seconds, nothing is done to the corresponding CEF entry. If the timer is less than 120 seconds, the NHRP process marks the corresponding CEF entry as “stale” but still usable. As soon as the router switches an IP packet using the “stale” entry, it triggers new NHRP resolution request, and eventually refreshes the corresponding NHRP entry as well as CEF entry itself. If no packet hits the “stale” CEF entry, the NHRP mapping will eventually time-out (since the router does not send any “refreshing” requests) and the corresponding CEF entry will become invalid. This will effectively tear down the spoke-to-spoke tunnel.
- Mappings can be static or dynamic.
- On the branch (spoke) routers, you must configure at least one static NHRP mapping in order to reach the NHS.
- Additionally, you would need to enable broadcast/multicast over the spoke-hub tunnel for routing protocols and other multicast data to work over the DMVPN tunnel.
- IP multicast/broadcast packets are supported only on tunnels with static mappings (the spoke-hub tunnels). Hubs are configured to enable NHRP to automatically add routers to multicast NHRP mappings. This is necessary to run routing protocols over the DMVPN mGRE tunnel. NHRP can only add a peer to the multicast mapping list when it receives an NHRP registration packet (only on NHRP NHSs).
- NHRP hold time is used to determine how long receiving routers should consider the cached entry information to be valid.
- NHRP registration requests are sent periodically (by default, one-third of the holdtime) to keep a NHRP registration mapping entry on the hub from timing out.
- Other NHRP options Cisco recommends to use are authentication and network-ID. NHRP authentication is used to authenticate the NHRP messages. All routers configured with NHRP within a DMVPN cloud must share the same authentication string.
NHRP Flags:
- authoritative: NHRP information was obtained from the Next Hop Server,
- implicit: Indicates that the information was learned not from an NHRP request generated from the local router, but from an NHRP packet being forwarded or from an NHRP request being received by the local router.
- negative: indicates that the requested NBMA mapping could not be obtained. When NHRP sends an NHRP resolution request it inserts an incomplete (negative) NHRP mapping entry for the address in the resolution request. This is to keep the router from triggering more NHRP resolution requests while this NHRP resolution request is being resolved and the IKE or IPsec tunnel created.
- unique: NHRP registration request packet had the "unique" flag set (on by default). NHRP mapping entry cannot be overwritten with a mapping entry that has the same IP address but a different NBMA address.
- registered: The mapping entry was created from receiving an NHRP registration request. The NHC must periodically send NHRP registration requests to keep these mappings from expiring.
- used:When data packets are process-switched and this mapping entry was used, the mapping entry is marked as used.
- router: NHRP mapping entries that are for a remote router.
- local: NHRP mapping entries that are for a network's local to this router (serviced by this router)
Topology
 |
| DMVPN topology |
Traffic flow example [VPC-4] --> [VPC-5]:
1) VPC-4 (192.168.160.10) is sending to VPC-5 (192.168.164.10) via defa GW: 192.168.160.1
2) Default GW is HUB (192.168.160.1) checks RIB
HUB# sh ip route 192.168.164.10
Routing entry for 192.168.164.0/24
Known via "static", distance 1, metric 0
* 172.16.1.2
HUB# sh ip route 172.16.1.2
Routing entry for 172.16.1.0/24
Known via "connected", distance 0, metric 0 (connected, via interface)
* directly connected, via Tunnel1
HUB# sh int tun 1 | inc GRE
Tunnel protocol/transport multi-GRE/IP
HUB#
HUB# show ip nhrp 172.16.1.2
172.16.1.2/32 via 172.16.1.2
Tunnel1 created 04:02:12, expire 01:58:12
Type: dynamic, Flags: registered nhop
NBMA address: 10.10.10.5
HUB#
HUB# show ip route 10.10.10.5
% Subnet not in table
HUB# show ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
Known via "static", distance 1, metric 0, candidate default path
* 10.10.10.2
3) Router Spoke 1 (10.10.10.5) receive ecrypter packet on int Gi 0/0,
After decryption 172.16.1.2, destination is Gi0/1 (192.168.164.1).
Configs
IPSec is the same for all DMVPN routers (HUB, Spoke1, Spoke2) and it is applied to interface Tunnel.
! IPSEC - IKE phase 1 (HAGLE)
crypto isakmp policy 1
encryption 3des
hash md5
authentication pre-share
group 2
!
crypto isakmp key isakmp1234 address 0.0.0.0 !! accept from any source to accommodate also dynamic spokes
!
! IPSEC - IKE phase 2 (IPsec transform)
crypto ipsec transform-set TS esp-3des esp-md5-hmac
mode tunnel
!
crypto ipsec profile PROTECT-GRE
set security-association lifetime seconds 86400
set transform-set TS
!
Inet (emulation) routers use EIGRP:
hostname inetx!
router ei 1
network 10.0.0.0
network 1.0.0.0
no au
!
Hub router
hostname HUB
!
interface GigabitEthernet0/0
description -=to Inet
ip address 10.10.10.1 255.255.255.252
!
interface GigabitEthernet0/1
description -=to LAN
ip address 192.168.160.1 255.255.255.0
!
interface Tunnel1
description -= DMVPN Tunnel
ip address 172.16.1.1 255.255.255.0 !! Select a private IP subnet for the tunnels
!
tunnel protection ipsec profile PROTECT-GRE !! encrypts the traffic passing through this tunnel using ipsec
tunnel source GigabitEthernet0/0 !! source of the tunnel is the WAN interface
tunnel mode gre multipoint !! designates the tunnel as a mGRE tunnel
!
ip nhrp authentication nhrp1234 !! (optional) authentication used for updates between the routers
ip nhrp network-id 1 !! (mandatory) network identification that has to be the same on all the routers
!
ip nhrp map multicast dynamic !! Enables multicast across the tunnel, the hub replicates multicast packets to all spokes registered via NHRP
! Add spoke to multicast list upon registration
!
!
!
ip mtu 1440 !! Reduce the MTU to allow extra overhead from mGRE and IPSEC
load-interval 30
keepalive 5 10
!
!
! Configure static routing on HUB (dynamic routing is recommended for larger networks)
ip route 192.168.164.0 255.255.255.0 172.16.1.2 !! The remote LAN can be reached via the remote tunnel IP
ip route 192.168.161.0 255.255.255.0 172.16.1.3 !! The remote LAN can be reached via the remote tunnel IP
ip route 0.0.0.0 0.0.0.0 10.10.10.2
!
Spoke-1
hostname SPOKE-1ip nhrp registration no-unique -
!
interface GigabitEthernet0/0
no sh
description -=to Inet
ip address 10.10.10.5 255.255.255.252
!
interface GigabitEthernet0/1
no sh
description -=To LAN
ip address 192.168.164.1 255.255.255.0
!
interface Tunnel1
description -= DMVPN Tunnel
ip address 172.16.1.2 255.255.255.0
!
tunnel protection ipsec profile PROTECT-GRE
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
!
ip nhrp authentication nhrp1234
ip nhrp network-id 1
!
ip nhrp map multicast dynamic !! Enables forwarding of multicast traffic across the tunnel.
ip nhrp map multicast 10.10.10.1 !! Send multicast to the Hub only. Hub will receive all multicast (e.g EIGRP updates) and then send out updates to all the Spoke routers.
! This step is required on multipoint GRE tunnels and not required on point-point RE tunnels.
ip nhrp map 172.16.1.1 10.10.10.1 !! required to be static map the tunnel IP of the HUB to the WAN IP of the HUB (not needed with PVC)
ip nhrp nhs 172.16.1.1 !! configures NHRP client with the IP address of its NHRP server (Next Hop Server)
ip nhrp registration no-unique * !! if a NHRP map is done for this IP another one will not be allowed
!
ip mtu 1440
load-interval 30
keepalive 5 10
!
ip route 192.168.160.0 255.255.255.0 172.16.1.1
ip route 192.168.161.0 255.255.255.0 172.16.1.3
ip route 0.0.0.0 0.0.0.0 10.10.10.6
!
Если изменится внешний адрес spoke-маршрутизатора и этой команды не будет, то hub-маршрутизатор не обновит свою базу nhrp из-за ошибки: unique address registered already
Spoke-2
hostname SPOKE-2
interface GigabitEthernet0/0
no sh
description -=to INET
ip address 10.10.10.9 255.255.255.252
!
interface GigabitEthernet0/1
no sh
description -=To LAN
ip address 192.168.161.1 255.255.255.0
!
interface Tunnel1
description -= DMVPN Tunnel
ip address 172.16.1.3 255.255.255.0
!
tunnel protection ipsec profile PROTECT-GRE
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
!
ip nhrp authentication nhrp1234
ip nhrp network-id 1
!
ip nhrp map multicast dynamic
ip nhrp map multicast 10.10.10.1
ip nhrp map 172.16.1.1 10.10.10.1
ip nhrp nhs 172.16.1.1
ip nhrp registration no-unique
!
ip mtu 1440
load-interval 30
keepalive 5 10
!
ip route 192.168.160.0 255.255.255.0 172.16.1.1
ip route 192.168.164.0 255.255.255.0 172.16.1.2
ip route 0.0.0.0 0.0.0.0 10.10.10.10
!
Verification
# View traffic on Router "inet2" int Gi0/1
# VPC-4 (192.168.160.10) => HUB => 10.10.10.2->1.1.1.2->10.10.10.5 => Spoke1 => VPC-5 (192.168.164.10)
VPC-4> show ip
IP/MASK : 192.168.160.10/24
GATEWAY : 192.168.160.1
MTU : 1500
VPC-4>
VPC-4> ping 192.168.164.10 -t
84 bytes from 192.168.164.10 icmp_seq=1 ttl=62 time=17.546 ms
84 bytes from 192.168.164.10 icmp_seq=2 ttl=62 time=11.317 ms
Wireshark:
 |
| DMVPN with IPSEC Tunnel mode + ESP |
Same traffic without ESP:
 | |
|
Troubleshooting
!
show crypto session
show crypto isakmp sa ! if you see QM_IDLE, it's good.
show crypto isakmp policy
show crypto isakmp diagnose error
show crypto ipsec sa
show crypto ipsec sa | inc crypt: ! if you are seeing packet being encrypted and decrypted, it's good.
show dmvpn
show dmvpn int tunnel 1 detail
show ip nhrp
show ip nhrp br
show ip nhrp nhs detail
show adjacency 172.16.1.3
show ip cef 172.16.1.3 internal
!
show run interface tunnel 1
! disable cef before debug ip packet !!!!!
debug ip packet
debug tunnel
debug nhrp
debug nhrp packet
debug cry isakmp
debug cry ipsec
undebug all
!
clear crypto isakmp
clear crypto sa
!
Examples:
IPSEC
HUB# show crypto session
Crypto session current status
!
Interface: Tunnel1
Session status: UP-ACTIVE
Peer: 10.10.10.9 port 500
Session ID: 0
IKEv1 SA: local 10.10.10.1/500 remote 10.10.10.9/500 Active
IPSEC FLOW: permit 47 host 10.10.10.1 host 10.10.10.9
Active SAs: 2, origin: crypto map
!
Interface: Tunnel1
Session status: UP-ACTIVE
Peer: 10.10.10.5 port 500
Session ID: 0
IKEv1 SA: local 10.10.10.1/500 remote 10.10.10.5/500 Active
IPSEC FLOW: permit 47 host 10.10.10.1 host 10.10.10.5
Active SAs: 2, origin: crypto map
!
HUB#
HUB# show crypto isakmp sa
!
IPv4 Crypto ISAKMP SA
dst src state conn-id status
10.10.10.5 10.10.10.1 QM_IDLE 1006 ACTIVE
10.10.10.1 10.10.10.9 QM_IDLE 1007 ACTIVE
HUB#
HUB#show crypto isakmp policy
Global IKE policy
Protection suite of priority 1
encryption algorithm: Three key triple DES
hash algorithm: Message Digest 5
authentication method: Pre-Shared Key
Diffie-Hellman group: #2 (1024 bit)
lifetime: 86400 seconds, no volume limit
HUB#
HUB# show crypto ipsec sa | inc crypt:|current_peer
current_peer 10.10.10.9 port 500
#pkts encaps: 8, #pkts encrypt: 8, #pkts digest: 8
#pkts decaps: 8, #pkts decrypt: 8, #pkts verify: 8
current_peer 10.10.10.5 port 500
#pkts encaps: 7, #pkts encrypt: 7, #pkts digest: 7
#pkts decaps: 7, #pkts decrypt: 7, #pkts verify: 7
HUB#
DMVPN
HUB# show dmvpn | be Interface
Interface: Tunnel1, IPv4 NHRP Details
Type:Hub, NHRP Peers:2,
!
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 10.10.10.5 172.16.1.2 UP 05:40:08 D
1 10.10.10.9 172.16.1.3 UP 04:25:17 D
!
HUB#
HUB# show ip nhrp
172.16.1.2/32 via 172.16.1.2
Tunnel1 created 05:40:37, expire 01:39:47
Type: dynamic, Flags: registered nhop
NBMA address: 10.10.10.5
172.16.1.3/32 via 172.16.1.3
Tunnel1 created 04:25:46, expire 01:54:29
Type: dynamic, Flags: registered nhop
NBMA address: 10.10.10.9
HUB#
IP CEF
HUB#show adjacency 172.16.1.3
Protocol Interface Address
IP Tunnel1 172.16.1.3(10)
!
HUB#show ip cef 172.16.1.3 internal
172.16.1.3/32, epoch 0, flags [att], refcnt 6, per-destination sharing
sources: Adj, RR
subblocks:
Adj source: IP midchain out of Tunnel1, addr 172.16.1.3 0F521D98
Dependent covered prefix type adjfib, cover 172.16.1.0/24
1 RR source [no flags]
ifnums:
Tunnel1(9): 172.16.1.3
path list 0D7964C4, 2 locks, per-destination, flags 0x49 [shble, rif, hwcn]
path 0D7951AC, share 1/1, type adjacency prefix, for IPv4
attached to Tunnel1, IP midchain out of Tunnel1, addr 172.16.1.3 0F521D98
output chain:
IP midchain out of Tunnel1, addr 172.16.1.3 0F521D98
IP adj out of GigabitEthernet0/0, addr 10.10.10.2 0E7279C0
HUB#
Some issued during lab configuration:
SPOKE-1#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
10.10.10.1 10.10.10.5 MM_NO_STATE 0 ACTIVE
10.10.10.1 10.10.10.5 MM_NO_STATE 0 ACTIVE (deleted)
SPOKE-1#
workaround: reload routers
The following four modes are found in IKE main mode
- MM_NO_STATE* – ISAKMP SA process has started but has not continued to form (typically due to a connectivity issue with the peer)
- MM_SA_SETUP* – Both peers agree on ISAKMP SA parameters and will move along the process
- MM_KEY_EXCH* – Both peers exchange their DH keys and are generating their secret keys. (This state could also mean there is a mis-matched authentication type or PSK, if it does not proceed to the next step)
- MM_KEY_AUTH* – ISAKMP SA’s have been authenticated in main mode and will proceed to QM_IDLE immediately.
The following three modes are found in IKE aggressive mode
- AG_NO_STATE** – ISAKMP SA process has started but has not continued to form (typically do to a connectivity issue with the peer)
- AG_INIT_EXCH** – Peers have exchanged their first set of packets in aggressive mode, but have not authenticated yet.
- AG_AUTH** – ISAKMP SA’s have been authenticated in aggressive mode and will proceed to QM_IDLE immediately.
The following mode is found in IKE Quick Mode, phase 2
- QM_IDLE*** – The ISAKMP SA is idle and authenticated
sh crypto session
This command will give you a quick list of all IKE and IPSec SA sessions.
Some of the common session statuses are as follows:
- Up-Active – IPSec SA is up/active and transferring data.
- Up-IDLE – IPSsc SA is up, but there is not data going over the tunnel
- Up-No-IKE – This occurs when one end of the VPN tunnel terminates the IPSec VPN and the remote end attempts to keep using the original SPI, this can be avoided by issuing crypto isakmp invalid-spi-recovery
- Down-Negotiating – The tunnel is down but still negotiating parameters to complete the tunnel.
- Down – The VPN tunnel is down.