NetPro Ask The Expert Yusuf Bhaiji on CCIE Security

Here's a very insightful thread from Cisco's NetPro Conections Forum. Yusuf Bhaiji, Program Manager for the Cisco CCIE Security certification and lab proctor, takes time to answer questions about the lab exam. The answers to most questions are mostly already available in the Lab Blueprint. But the Q&A helps to clarify some questions a CCIE candidate might have. Here are some of Yusuf's responses to questions which I believe is useful in knowing what to expect from the CCIE lab exam.

"As mentioned in the new blueprints, the new exam is heavily focused on Security technologies only, and routing functions are tested on Security appliances only. Advance Routing features such as filtering, summarization etc are no longer core objectives, and is tested mostly on written exam. "

"Yes, since VPN3000 concetrator is announced EOS, it is very likely to be removed from the CCIE lab exam. We are presently working on this and will make an announcement when a decision has been made. Meanwhile, it will continue to appear in the exam. "

"Yes, all routing & switching is pre-configured on all devices except the security appliances (i.e. PIX/ASA, VPN3k, IDS). Candidates are required to configure everything on security appliances."

"1) PIX/ASA will be running version 7.2.x (Q: Can you tell us what exact release is currently running on the exam for ASA, is it 7.2(1)? , or at least is it 7.2(X)?)
2) IPS version 5.1.x (Q:For IPS, is the lab running 5.1? )
3) NAC Framework can be tested on other devices such as the Switch or VPN3k etc (Q:Regarding NAC, on IOS it is not supported on 12.2(T) the current IOS, how will the lab test NAC then ?)
4) Both, promiscuios and inline. (Q:Do we still have to prepare for promiscuous mode IPS (or IDS) deployment or line and inline-vlan pair only? )"

"1) It depends on the requirements & restrictions, but in most cases, there is no penalty for over-configuration. (Q:Are we penalized for over-configuration on the CCIE Security Lab?)
2) Again, it depends on the question's requirements & restrictions. There is no golden rule that you can apply. (Q:can we configure more 'generic' ACLs ? I mean in the security LAB are ACLs supposed to be as specific as possible)
Pls read the questons carefully, and if unclear, pls ask the proctor for clarification. "

"The grading is entirely dependant on the proctor. Having said that, we also use 'automated tools' to expedite some of the repetitive tasks in grading and checking same stuff on all devices, but ultimately, it is the proctor who decides to judge if the answer is correct and/or reward the points. "

"Troubleshooting is most likely within the pre-configuration as security appliances (PIX/ASA, VPN3k, IDS) does not have any configuration except the basic (hostname, enable pwd, etc). "

"...all aspects of routing on security appliance can appear on the exam. But pls note, the emphasis is NOT routing, but these will appear merely to complete an exercise. "

"Troubleshooting is mainly focused on FUNCTIONALITY. For e.g. there will be a broken scenario (security context) e.g. IPsec LAN-to-LAN is pre-configured but NOT working. You will require to identify and fix the end-to-end thing and ensure it is working. The issue can be related to IPsec config or even non-IPsec config within... "

My Favorite:
"No Hints (from the proctor) :) The lab proctor is available to clarify the requirements only. He cannot offer you any hints. "

Read the rest of the conversation on this link.

Router: Debug Crypto Condition command

When your router has more than one vpn peer configured, troubleshooting one of the problematic crypto peer using debug crypto isakmp and debug crypto ipsec can get you crosseyed in no time. You'll feel like the console is doing a DOS on your brain.

In Cisco IOS Software Release 12.3(2)T, you can use the debug crypto condition command to filter debug output to a specific peer. You can use many criteria to customize the output, including peer IP, SPI, connid, flowid, etc. This filter causes the router to show only the crypto messages for the peer you are troubleshooting.

For example, first define a condition with a peer IP before enabling debug crypto:
debug crypto condition peer ipv4 2.2.2.2
debug crypto isakmp
debug crypto isakmp packet <---helpful hidden command debug crypto ipsec
To identify which debug conditions are active:
sh crypto debug-condition

Lab: RTBH

This is something I setup in my desktop lab (Dynamips) based on the REMOTELY TRIGGERED BLACK HOLE FILTERING—DESTINATION BASED AND SOURCE BASED white paper available from Cisco website.

Configurations
Target
Edge
Trigger
RR-BGP
PE1
PE2
Smurf

In the diagram, the provider network is enclosed in the blue area. The Target router is the victim and Smurf is the attacker network. Let's say we're seeing massive amount of traffic coming from Smurf network with destination as the Target router. We identify these as a dos attack on the Target network so as Service Provider we must: 1. Protect our customer and 2. Protect the provider network.
RTBH provides a method of telling the provider edge routers to block an identified traffic just by adding the source or destination address to a sort of blacklist. These blacklist are actually static routes added to the Trigger router that are propagated throughout the network via iBGP. These routes points to a null interface as its next-hop, so effectively dropping the bogus traffic.

Lab: Integrated DMVPN and EZVPN with IPSec Stateful Failover

Here's something I setup in my desktop "lab" using DYNAMIPS. I created two DMVPN hubs also acting as a EZVPN server with Stateful Failover. Since both DMVPN and EZVPN configuration is using 0.0.0.0 address to map to the pre-shared key, a ISAKMP Profile is configured to differentiate the DMVPN spokes and the EZVPN clients.

Previously, if your router talk to a combination of remote spokes and vpn client (software or ezvpn remote) that have dynamic ip addresses, your only option is to use certificates for your IKE authentication. If you use pre-shared keys, you cannot distinguish which IKE peers should have extended authentication and which one should not use it. Well that was before ISAKMP Profile was available. ISAKMP Profile should be used for any router with two or more IPSec connections that requires different phase 1 parameters for different sites (for example, configuring site-to-site and remote access on the same router). You can configure to authenticate one peer using certificates while another peer is authenticated using pre-shared keys.

Here are the configs.
Hub1 Configuration
Hub2 Configuration
EZVPN Remote Client Configuration
DMVPN Spoke1 Configuration
DMVPN Spoke2 Configuration

You can refer to Cisco's Deployment Guide for more detailed info.

Pix/ASA: DNS rewrite and Packet U-Turns part II

Network

Internet----(outside)pix/asa(inside)----client
                    (dmz)
                      I
                      I
                    server

Now let's consider a pix/asa with three interfaces: inside, outside and dmz. Client PC in the inside network cannot access the server in the dmz eventhough we have the correct nat configuration in the firewall. We soon found out that the client PC is resolving server's hostname to its public ip. Problem is the pix/asa will forward the packet to the outside interface because the destination in the packet header is the server's public ip.

Solution 1:
DNS doctoring to the rescue

global (outside) 1 interface
nat (inside) 1 192.168.100.0 255.255.255.0
static (inside,dmz) 192.168.100.0 192.168.100.0 netmask 255.255.255.0
static (dmz,outside) 172.20.1.10 10.10.10.10 netmask 255.255.255.255 dns

By adding the dns keyword in the static nat for the server, we are telling the pix to rewrite the dns query response so that the client pc will resolve the server's hostname to its private ip. So then the pix will correctly route to the dmz interface when client PC sends a packet to the server.
Make sure that you have DNS inspection configured for DNS Doctoring to work.

policy-map global_policy
class inspection_default
...
inspect dns

Solution 2:
Another way to get around the problem is to configure the pix to do a nat translation for packets from inside hosts going to the server in the dmz. But instead of natting the source address in the packet header, we tell it to nat the destination address.

!
! Nat for inside-to-outside traffic
!
global (outside) 1 interface
nat (inside) 1 192.168.100.0 255.255.255.0
!
! Nat for inside-to-dmz traffic to server
!
static (inside,dmz) 192.168.100.0 192.168.100.0 netmask 255.255.255.0
!
! Nat for dmz-to-outside traffic to server
!
static (dmz,outside) 172.20.1.10 10.10.10.10 netmask 255.255.255.255
!
! Destination Nat for the server's IP
!
static (dmz,inside) 172.20.1.10 10.10.10.10 netmask 255.255.255.255


More detailed info from the Cisco sample configuration.

Pix/ASA: DNS rewrite and Packet U-Turns part I

Topology

                                    I--Client
Internet----(Outside)ASA(Inside)----I
                                    I--Server

Problem

An internal host cannot reach the a public server in the DMZ arm of the pix/asa. The reason is because the public server's hostname is resolved by DNS to its public ip address and the pix/asa will not route a packet to the outside interface then u-turn it back to its inside interface.

Solution 1: We tell the pix/asa to hack into the DNS query response such that the client host will resolve the server's hostname to it's private ip address.
Make sure that you have DNS inspection configured for DNS Doctoring to work.

policy-map global_policy
class inspection_default
...
inspect dns

! Nat/Global for outbound traffic
!
global (outside) 1 interface
nat (inside) 1 192.168.100.0 255.255.255.0
!
! Static nat for inbound traffic to the server.
! The DNS keyword at the end enables DNS Doctoring
!
static (inside,outside) 172.20.1.10 192.168.100.10 netmask 255.255.255.255 dns
!

It beats going to each and every workstation on your network and adding the server's hostname/ip address in their etc/hosts file.

Solution 2.
On pre-7.2 verion of the security appliance software, the pix/asa will not forward clear (unencrypted) traffic back the same interface where it was recieved from. On 7.1, it will allow the packet to take a U-turn but only for encrypted traffic.
So if the firewall is running 7.2 software, we can get around these limitations and allow the client host to access the server using the resolved public ip address.

! To allow packet U-turn
!
same-security-traffic permit intra-interface
!
! global statement for outbound traffic
!
global (outside) 1 interface
!
! global statement for intra-interface traffic
!
global (inside) 1 interface
nat (inside) 1 192.168.100.0 255.255.255.0
!
! Static nat for inbound traffic
!
static (inside,outside) 172.20.1.10 192.168.100.10 netmask 255.255.255.255
!
! To map the public ip of the server to its private ip
! for intra-interface traffic.
!
static (inside,inside) 172.20.1.10 192.168.100.10 netmask 255.255.255.255
!

For a more detailed explanation, read this sample configuration from Cisco.

Router: Zone-based Firewalls part III

Now let's look at an example where we have three zones: inside, outside and dmz. There are servers in the DMZ zone that needs to be accessible from the inside and outside network. We will restrict Outside to DMZ traffic only to ip 192.1.1.1 via http.

! we define the zones
!
 zone security Inside
 zone security Outside
 zone security DMZ
!
! we apply the zones to the interfaces
!
interface FastEthernet0/0
 ...
 zone-member security Inside
!
interface FastEthernet0/1
 ...
 zone-member security DMZ
!
interface Serial0/0.100 point-to-point
 description Link to the Internet
 ...
 zone-member security Outside
!
! Match any traffic going to the Webserver
!
access-list 199 permit ip any host 192.1.1.1
!
! match traffic to be inspected
!
class-map type inspect insp-traffic
 match protocol http
 match protocol icmp
 match protocol tcp
!
! match http traffic going to the webserver
!
class-map type inspect match-all http_traffic
 match access-group 199
 match protocol http
!
class-map type inspect match-all toDMZfromInside
 match access-group 199
 match class-map insp-traffic
!
class-map type inspect match-any DNS
 match protocol dns
!
policy-map type inspect out_traffic
 class type inspect insp-traffic
  inspect
!
policy-map type inspect toDMZfromInsideTraffic
 class type inspect toDMZfromInside
  inspect
!
policy-map type inspect webtraffic
 class type inspect http_traffic
  inspect
!
policy-map type inspect DNSTraffic
 class type inspect DNS
  inspect
!
! security policy for inside to outside traffic
!
zone-pair security InsideToOutside source Inside destination Outside
service-policy type inspect out_traffic
!
! security policy for outside to dmz traffic
!
zone-pair security OutsideToDMZ source Outside destination DMZ
service-policy type inspect webtraffic
!
! security policy for inside to dmz traffic
!
zone-pair security InsideToDMZ source Inside destination DMZ
service-policy type inspect toDMZfromInsideTraffic
!
! security policy for dmz to outside traffic
!
zone-pair security DMZToOutside source DMZ destination Outside
service-policy type inspect DNSTraffic
!

Notice that there is no security policy for traffic going from Outside to Inside or from DMZ to Inside. This configuration will drop any traffic going to the Inside network.

Zone-based Firewall is modular making the configuration much more easier to read. You can add more zones or you can add more interfaces to each zone without cluttering the configuration and thus easing troubleshooting. It removes the firewall's dependency on access-lists and allows you to configure one policy for any given traffic.

For more info, check out Cisco documentation (of course!) and discussion slide on zone-based firewall and the excellent ebook Deploying Zone-Based Firewalls (Digital Short Cut) from Ivan Pepelnjak.

Router: Zone-based Firewalls part II

In the first installment of this document, I showed you a simple two-zone configuration where inside users have unrestricted access to the outside zone.

If you need to limit the services that the inside users are allowed to access, we have to define the traffic classes that will later be used in the policy-map commands to define the desired firewall policy. The traffic classes are defined with the class-map command augmented with the type inspect keyword.

!
! These are the Class maps to define
! the outgoing traffic that are permitted.
!
class-map type inspect match-any MailAndDNS
match protocol dns
match protocol smtp
match protocol pop3
class-map type inspect match-all ISP_Traffic
match class-map MailAndDNS
match access-group name ISPServers
class-map type inspect match-any InternetTraffic
match protocol http
match protocol ftp
match protocol icmp
match protocol https
!
!
policy-map type inspect InsideToOutside
!
! Action for defined class-map is inspect
!
class type inspect ISP_Traffic
 inspect
class type inspect InternetTraffic
 inspect
!
! Action for all other traffic are dropped and logged
!
class class-default
drop log
!
! we define the zones
!
zone security Inside
zone security Outside
!
! we define the security policy
!
zone-pair security InsideToOutside source Inside destination Outside
service-policy type inspect InsideToOutside
!
! we apply the zones to the interfaces
!
interface FastEthernet0/0
...
zone-member security Inside
!
interface Serial0/0.100 point-to-point
description Link to the Internet
...
zone-member security Outside
!
ip access-list extended ISPServers
permit ip any host 192.1.1.1
permit ip any host 192.1.1.2

This time we used the action drop and log for the class class-default.
With this configuration, outbound traffic are restricted to those services defined in the class-maps. Matching traffic are permitted and inspected while those that does not match are dropped. There is still no security policy defined for inbound traffic (outside-to-inside) so it will be dropped by the router.

Router: Zone-based Firewalls part I

A new configuration enhancement has been introduced in IOS 12.4(6)T called Zone-based policy firewall. Rather than configuring multiple access-lists to filter traffic between multiple router interfaces, you follow the zone-based design and only have to specify the traffic permitted between zones. Zone-based policy firewall also adds more granularity to inspection policies comapared to CBAC.

Here are some notes:

1. The zone-based policy firewall can coexist with Cisco IOS Firewall stateful inspection. You can still use the ip inspect command on interfaces that are not members of security zones.
   
2. Traffic can never flow between an interface assigned to a zone and an interface without a zone assignment.
   
3. The default interzone policy is to drop all traffic unless specified otherwise in the zone-pair configuration command.
   
4. The router never filters the traffic between interfaces in the same zone.
   
5. If two interfaces are not in zones, traffic flows freely between them.
   
6. The zone-member command does not protect the router itself (traffic to and from the router is not affected) unless you configure the zone pairs using the predefined self zone.
   

Examples are available on this slide.

Here's a simple example.

! Create the zones. for this example we only have two zones
!
zone security Inside
 description This is the Internal network

zone security Outside
 description This Internet zone
!
! Using policy-map, we specify the action to do
! on the traffic matching the class-maps
! For this example, we use the default class
! class-default = match all
!
policy-map type inspect InsideToOutside
 class class-default
 inspect
!
! Apply the policy to traffic between a pair of zones
!
 zone-pair security InsideToOutside source Inside destination Outside
 service-policy type inspect InsideToOutside
!
! Assign the interfaces to each zones
!
interface FastEthernet0/0
 zone-member security Inside
!
interface Serial0/0/0.100 point-to-point
 zone-member security outside

This is a simple two-zone configuration where a security policy is defined for outbound traffic (inside-to-outside). Inbound traffic will be dropped since there is no security policy defined for outside-to-inside traffic.

Router: Surfing without Split-tunnelling

Scenario: Users connect remotely via Cisco VPN client. They connect to your router. They need to access the Internet while logged-in but you don't want to configure split-tunnelling. You want the VPN client to access the internet thru the router and not by split-tunneling so that you can later enable url-filtering or use audit-trail to monitor their browsing activities.
Config:

username cisco password cisco123
aaa new-model
!
!
aaa authentication login userauthen local
aaa authorization network groupauthor local
!
crypto isakmp policy 3
encr 3des
authentication pre-share
group 2
!
crypto isakmp client configuration group RAS
key cisco123
domain cisco.com
pool ippool
!
!
crypto ipsec transform-set myset esp-3des esp-md5-hmac
!
crypto dynamic-map dynmap 10
set transform-set myset
!
!
crypto map clientmap client authentication list userauthen
crypto map clientmap isakmp authorization list groupauthor
crypto map clientmap client configuration address respond
crypto map clientmap 10 ipsec-isakmp dynamic dynmap
!
! User's traffic will be redirected to this loopback interface
!
interface Loopback0
ip address 1.1.1.1 255.255.255.0
ip nat inside
!
interface Ethernet0/1
ip address 192.168.20.1 255.255.255.0
ip nat inside
!
interface Ethernet0/0
...
ip nat outside
ip policy route-map redir

crypto map clientmap
!
ip local pool ippool 192.168.1.100 192.168.1.200
ip nat inside source list NAT interface Ethernet0/0 overload
!
ip access-list extended NAT
deny ip 192.168.20.0 0.0.0.255 192.168.1.0 0.0.0.255
permit ip 192.168.20.0 0.0.0.255 any
permit ip 192.168.1.0 0.0.0.255 any
!
access-list 101 permit ip 192.168.1.0 0.0.0.255 any
!
! This route map redirects vpn client traffic to Loopback0
!
route-map redir permit 10
match ip address 101
set interface Loopback0



On the pix/asa, check this sample configuration from Cisco for a similar "vpn-on-a-stick" setup.

Pix/ASA: Packet Tracer

This new feature is very useful in troubleshooting or verifying your configuration. It is available in ASDM and CLI.
Syntax:

packet-tracer input [src_int] protocol src_addr src_port dest_addr dest_port [detailed] [xml]

It is a good complement to the capture command when troubleshooting. To use it, you just enter the source and destination ip and port address and the packet-tracer will provide detailed information on how the security appliance will process that packet and will also show you why it will fail based on your firewall's configuration.

Example from the command reference:
To enable packet tracing from inside host 10.2.25.3 to external host 209.165.202.158 with detailed information, enter the following:

hostname# packet-tracer input inside tcp 10.2.25.3 www 209.165.202.158 aol detailed

Learn more about the packet-tracer tool from the Quick Learning Module available from Cisco.

Pix/ASA: View interface status

One of the first command that you usually learn on IOS is the useful command 'show ip interface brief'. This command will allow you to see the list of available interfaces on the router and their status.

Do you know that the pix have an equivalent command? It's easy to miss it because those folks at PixOS Development have rearrange the words a bit. Well the command on the pix or ASA is 'show interface ip brief'.

hostname# show interface ip brief

Interface                  IP-Address      OK? Method  Status                Protocol

Control0/0                 127.0.1.1       YES CONFIG  up                    up

GigabitEthernet0/0         209.165.200.226 YES CONFIG  up                    up

GigabitEthernet0/1         unassigned      YES unset   administratively down down

GigabitEthernet0/2         10.1.1.50       YES manual  administratively down down

GigabitEthernet0/3         192.168.2.6     YES DHCP    administratively down down

Management0/0              209.165.201.3   YES CONFIG  up

Nifty eh. It's a good thing that the Pix/ASA commands are slowly becoming similar in syntax as IOS commands. So that we don't have to go through the confusion again like learning the difference between the conduit and the access-list command.

Pix/ASA: Inserting Access-list Entries

My boss didn't know that on the Pix you can insert an access-list in between previous entries. He have always relied on the ASDM to do that.
As showed to him, you can do it via CLI and this has always been possible on the pix since version 6.3.
The first step is to do a show access-lists.

PIX# show access-list
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096) alert-interval 300
access-list ACLIN; 3 elements
access-list ACLIN line 1 extended permit tcp any host 172.31.0.10 eq www (hitcnt=0)
access-list ACLIN line 2 extended permit tcp any host 172.31.0.10 eq ftp (hitcnt=0)
access-list ACLIN line 3 extended permit icmp any host 172.31.0.10 (hitcnt=0)

As you can see, it will show you the sequence number of each access-list entries. You can then use this numbers to insert your new access-list anywhere. Creating a new 'access-list ACLIN line 1' will push the existing first entry down.

PIX# conf t
PIX(config)# access-list ACLIN line 1 extended deny tcp 10.0.0.0 255.0.0.0 host 172.31.0.10 eq www
PIX(config)# access-list ACLIN line 2 extended deny tcp 172.16.0.0 255.255.0.0 host 172.31.0.10 eq www
PIX(config)# access-list ACLIN line 3 extended deny tcp 192.168.0.0 255.255.0.0 host 172.31.0.10 eq www
PIX(config)# ! Without specifying the line number, the PixOS will
PIX(config)# ! place the entry at the end of the access-list
PIX(config)# access-list ACLIN extended deny ip any any log
PIX(config)# show access-list
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096) alert-interval 300
access-list ACLIN; 7 elements
access-list ACLIN line 1 extended deny tcp 10.0.0.0 255.0.0.0 host 172.31.0.10 eq www (hitcnt=0)
access-list ACLIN line 2 extended deny tcp 172.16.0.0 255.255.0.0 host 172.31.0.10 eq www (hitcnt=0)
access-list ACLIN line 3 extended deny tcp 192.168.0.0 255.255.0.0 host 172.31.0.10 eq www (hitcnt=0)
access-list ACLIN line 4 extended permit tcp any host 172.31.0.10 eq www (hitcnt=0)
access-list ACLIN line 5 extended permit tcp any host 172.31.0.10 eq ftp (hitcnt=0)
access-list ACLIN line 6 extended permit icmp any host 172.31.0.10 (hitcnt=0)
access-list ACLIN line 7 extended deny ip any any log informational interval 300 (hitcnt=0)

Access-lists on the IOS also have sequence numbering so you can do the same thing on routers.

RTFM first !!!

I work with Cisco boxes and when faced with a task of configuring something unfamiliar to me, there is usually 5 options:

1. RTFM
2. Search Cisco website for a sample configuration
3. Search the Internet, hoping that a kind soul have posted a solution to what you are trying to achieve
4. Ask the experts in mailing lists, study groups and forums.
5. Ask my more senior colleagues.

Most of the time, I'll find myself on my own. But sometimes, I find bits and pieces of useful information in the web, selflessly shared online. I can usually get away by copying from Cisco documentation or Cisco configurations posted by some generous netizens. Copy, paste and ready to go!

I have been a beneficiary of this information pool for quite some time now and figured it's time I give back to the community. Expect to see here tips and tricks, sample configurations, workarounds and whatever I think would be useful to my fellow Cisco professionals.

Having said that, let me end my first post with a DISCLAIMER:

No Warranty of any kind is expressed or implied with respect to the information contained in the documents found in this blog! The information found here is compiled for the convenience of anyone looking for general guidelines, reference materials and peer advice. Use this information at your own risk!