Cisco ASA TLS handshake error

If you receive ASA with only DES/MD-5 encryption license then you might wish to upgrade your firewall to the stronger (3DES/AES) encryption license. Even after upgrade you can run on problems with connecting your Windows 7 Internet Explorer 8 or later to the ASA https interface (ASDM or WebVPN). Connections from ASDM Launcher application works fine, but direct connections from MS IE browser are failing. Debug of the connection will show you a couple of things:


- ASA sends TCP RST packet right after TCP handshake
- TLS debug shows (visible from ASA debug level logging) cipher negotiation errors and this is the root cause of our little problem.

Well, new IE browser doesn't want to connect to your ASA because ASA supports only des/md-5 and rc4/md-5 encryption while IE requires something a little bit better as a minimum for completing TLS handshake. Such poor encryption is the result of your ASA being shipped with weak encryption license by default. So, after encryption license upgrade issue:
show run all ssl

The command from above will discover what SSL/TLS cipher suits your ASA currently supports and if you don't see 3DES and AES encryption algorithms listed here type:
ssl encryption 3des-sha1 aes128-sha1 aes256-sha1

Now you should be able to connect with MS IE 8 or newer;)

Cisco IPS signature 5377/0 caveat

Today I received a support call about user's pc that cannot connect to the Internet suddenly, but able to talk with internal hosts. I've checked AIP-SSM log and found that this PC fired 5377/0 signature which triggers if "xp_cmdshell" (MS SQL Server stored procedure) word is found in HTTP request. It turned that this is one of our developer's PC and she fired this signature by simply searching Google for xp_cmdshell keyword:)


So, if you have SQL Server guys in your network create Event Action Rule Override to avoid breaking network communication for these people...


Regards,
Igor

Cisco ASA/PIX service policy common mistakes

This time I'll talk about one common mistake often made by novice ASA admins when they try to configure protocol inspection. I saw too many broken Internet connections caused by this one, so it would be good to say something about it...

Let see what happens when you incorrectly select traffic for application inspection:

Wrong config:

http-map pure_http_only
 strict-http action drop


policy-map AppLayerInspection
 class web-traffic
 inspect http pure_http_only


class-map http_traffic
 match access-list http_traffic


access-list http_traffic permit ip any any


service-policy AppLayerInspection interface outside

Million $ question is what the hell is wrong with this config. Let's take a closer look...
Policy-map "AppLayerInspection" enforces strict (by RFC definition) HTTP on traffic classified by the class-map "http_traffic". And there is nothing wrong with that. The goal of strict HTTP inspection is that firewall recognizes if someone tries to run traffic which is not real HTTP trough port 80. The only problem with the configuration above is that class-map selects traffic according to access-list "http_traffic" which says permit ip any any. So if we have in mind that interface policy always supersedes global policy let's examine what ASA will do with for example simple outbound ping traffic:

We send ICMP ECHO from inside host to the Internet. This packet is processed by ASA firewall's outside interface service policy before it is sent to the upstream ISP router and since ICMP ECHO is definitely not a HTTP traffic crafted by RFC rules it's dropped. And the same destiny will find all other non-pure HTTP traffic including DNS, SMTP, etc, etc. So, practically your users will be able to connect only to some HTTP sites and all other connections will be denied!

So, what we should do to correct this major issue before phones starts ringing in panic? Simple change our "http_traffic" access-list into:
access-list http_traffic permit tcp any any eq 80

Now the access-list will select only packets with destination port 80 for the RFC HTTP compliance check and all other traffic will be processed by ASA global policy.

DNS problems with Cisco VPN Client on Windows 7 32-bit

These days I'm receiving support calls from our customers which say that they are unable to connect to the intranet resources with Cisco VPN Client.


A little investigation brought me to discovery that it's only about Cisco IPSec VPN Clients installed on Windows 7 32-bit. 64-bit version of Win 7/Vista and WinXP are fine.

A little deeper investigation isolated the problem: I noticed that vpn client upon successful connection with the Cisco router or ASA firewall doesn't receive DNS servers specified in tunnel group (ASA) or client configuration group (IOS). So, actually you can connect to your intranet servers using IP addresses, but not with DNS names. As I wrote above it wasn't about just one case but several. This required an emergency testing in my mighty lab :) - my laptop with VM Ware and Windows 7 32-bit virtual machine:)

Finally I resolved the case: it's a bug in the VPN client (I found it on Cisco Bug Toolkit under CSCsq34291 name). Cisco IPSec VPN Client version 5.0.07.0290 or newer don't have this problem, so I advised my customers to upgrade their Cisco VPN Clients. Another workaround according to Cisco is to use more than one DNS server in the tunnel group (or client configuration group). They claim that this issue shows its ugly face when only one DNS server is configured to be pushed on the client. If you are big then you already have at least two internal DNS servers (your domain controllers for example), but if you are small single server shop then just enter some fake IP address for secondary DNS in client configuration as quick workaround until you upgrade all the clients.

How to tune Cisco AIP-SSM IPS module

Note: this article has been written for AIP-SSM 7.0.3(E4) OS image with Cisco IPS Manager Express 7.0.3


The company for which I work bought me:) Cisco AIP-SSM module since I asked for something to use for IPS/IDS purposes. AIP-SSM was logical choice since I replaced our "old" 2811 IOS firewall + IPS router with ASA 5520 since the router's CPU was melting with our 50 megs Internet pipe and about 60 connections / sec + IOS FW+IOS IPS+traffic shaping with dozens of CBWFQ policies nested + NAT of course;) ASA 5520 does the same job (except QoS) with 3-5 % of average CPU usage.

In this blog post I want to share my experience with tuning this AIP-SSM baby.

Ok first I had to upgrade the thing (that is install new OS image) because I wanted to use new fancy Cisco IPS Manager Express software which I think is great and easy to use. After that I configured policy-map on my ASA firewall to inspect only traffic destined for my Internet facing web and ftp server which serves for secondary purposes so if the IPS breaks something very few people will notice:) And then the first issue. Suddenly I couldn’t connect to the web server from Internet any more. WTF? IPS event log is empty, but somehow I had a feeling that my connection was denied by the IPS. Ok. What was my first mistake? Well, since my web server is published to the Internet with static NAT configured on my ASA I placed IPS policy-map onto its outside interface and in class-map I specified web server's NAT inside global address. This (I guess) confused the ASA which routed traffic matched by IPS class-map to some nasty black hole. Very strange. I didn't expected such behavior. I expected that in the case of wrong traffic definition or misplaced IPS policy-map ASA does not catch anything to send to the AIP-SSM module, but not to block that traffic. I didn't sniffed Cisco Bug Toolkit for it, but this smells me like some ASA OS bug. I run 7.0 OS of the ASA for now, so if you run some later versions of ASA OS it may behave differently. Whatever, since ASA order of operations for inbound traffic says that inbound packets are first NATed back to the inside local address and then processed by modular policy framework I moved my IPS policy-map little closer to me - onto inside interface and class-maped internal IP address of the web server. And it worked like a charm. IPS detected some nmap port scanning I tried against my web server, root ftp password logins, etc.

Next after that little trial period I decided to put my entire Internet communication under the AIP-SSM microscope since the module has enough capacity to inspect my whole Internet tube.

Now folks, let's go serious so here is the rule number one: never, I mean really NEVER place your brand new out-of-the box IPS device in IPS mode for the first time. No matter how good and accurate your IPS might be there are good chances that you’re going to get some false positives. In the case of Cisco AIP-SSM if those false positives are with risk rating score of 90 or more then Event Action Override (if enabled and left with default actions) will ban potential attacker from your network for one hour period. This "attacker" might be your CEO's workstation or even worse: your Exchange Server:)) That is the reason why for the first time (for me it was about 1 month) you should configure IPS policy-map to work in IDS mode. In this mode ASA firewall will send only copy of your traffic to the AIP-SSM module, so it will be able to only detect, but not really block potential attackers and kill legitimate traffic. Also in IPS module configuration under Event Action Override uncheck the 'Reset TCP Connection' action checkbox for the duration of this testing (learning) phase because default Event Action Override rule states that any fired event with risk rating of 90 or more will cause AIP-SSM to (among other restrictive actions) send TCP resets to the attacker in the case that attack was carried on TCP protocol which is mostly the case. TCP resets are sent even if the device works in IDS module causing potential false positives to disrupt legitimate traffic.

Now when you have your module in completely Monday-morning-emergency-NoInternetConnection-calls free :) mode you should start tuning your IPS.

Before you start analyzing events a slight configuration of AIP-SSM module is strongly advised to make your life a little bit easer:

1. Define OS Identifications for your valuable and most risky assets (Internet facing servers for example) on this way IPS will know which attacks are relevant to your infrastructure.
2. Define Event variables - this helps later in the tuning phase 'cause you will create rules (Event Action Filters) to catch false positive traffic and prevent blocking it. In the Event Action Filters it's much easier to say for example $internal for all of your internal subnets then to count all your internal address space in each filter. This will also give you great flexibility if you renumber your IP network in the future in which case you only need to follow this change in Event Action Variables while leaving filters intact. On the same way network/host/protocol object grouping in ASA firewall configuration works.
3. Target Value Rating - specify hosts on your network that needs the greatest level of protection. Doing this AIP-SSM will greatly increase risk rating score for the attacks tried against these critical hosts.
4. Clear "Test Global Correlation" check box to make IPS stops blacklisted attackers immediately even before they try something evil.
5. Turn off "Network Participation" - this may sounds selfish but I don't want my internal hosts firing some false positive alarms to be listed on some black lists.

After implementation of the steps above it's time to examine IPS logs.

Wait for a week and then look for events with the risk rating of 90 or more. After you isolate these events dig even further searching for actions taken such as: deniedAttacker not performed, etc. These "not performed" actions are not performed because the module operates in IDS mode. In IPS mode these actions would be performed, of course. Now each of isolated events (risk rating >= 90 with some restrictive actions taken) should be analyzed by whoising victim and attacker IP address, examining traffic flow, packet capture traffic in some hard-to-distinguish cases, etc. For this you should really know your network and what kind of traffic and protocols are normal and good and which are pure evil:). Simply said you must be able to distinguish good from evil or else you have great chances to miss something important. After you analyzed events you should create Event Action Rule for each of them. In Event Action Rules subtract only blocking actions while leaving logging actions active. Some authors advise to subtract even logging events but I like to have everything logged for just a case if I need some historical data for forensic work, etc. Of course with lot of false positives in it your log will become a reading nightmare, so in IPS Manager Express I can create nice event reports filtering only important events without false positives. For example I usually like to see all events generated for my critical servers (even false positives if there are only few, otherwise only medium or high severities). For other things like outbound traffic I usually display only medium and high risk events to check if some worms are trying to spread or for some client side vulnerabilities exploitation attempts and so on.

As I previously said you should monitor your IPS and analyze logs on the way I described above for some period of time let say one month or even more. It depends on how quickly you need IPS protection in place, traffic volume (more traffic more like hood for false positives). Don't give up of IDS mode too quickly. I made a mistake and left it in IDS mode for a couple of days only and wrongly concluded that all false positives generated by my network are covered with Event Action Rules, but I was wrong. Here is the story: since we are small ISP for daughter firms inside our holding company one of our customers informed me by panic call:) that they don't have Internet connection anymore. Upon the call I was assumed that my new IPS has something with it:) and I was right. Signature ID 5930 (Generic SQL Injection) was fired for the traffic from public IP address of this customer. Risk Rating of this event was more then 90 and this customer was blocked for Internet access automatically. Normally some paranoid rookie admin would simply assume that IPS was right and someone from this network tried to SQL injection some web application on the Internet. But not me:) first I whoised the victims IP address and discovered that it is owned by Google:) Then I asked my DNS for RR record in hope that this IP has RR record. And bingo:) it was Google Analytics. So it was perfectly legitimate traffic since server from my customer updated some web statistics to Google Analytics and my IPS found SQL 'union select' statements in HTTP requests. I immediately created Event Action Rule subtracting blocking actions for such signature if "malicious" traffic flows from inside to outside. Now my IPS shall not disturb Google Analytics but it will prevent some SQL injection attacks against my Internet facing web applications.

Also there are a couple of noisy sweep signatures which normally should fire in the event of port or IP range scanning attempts but the ugly truth is that they fires for many multi connection applications. As you know some applications will try to connect to different hosts one by one (like network printer manager apps) which look like port scanning. It is completely safe to disable those signatures and leave port scanning and worm propagation detection to the Anomaly Detection feature of Cisco AIP-SSM.

And of course it's always good practice to test your IPS implementation against couple of known application exploits. For this I recommend Nessus scanner and Metasploit Framework combination. It's just to see if the IPS really works. Don't fool yourself that you will be able to test how effective your IPS is against tools like Metasploit framework because no way you can try all exploits and all the ways of tunneling/sneaking them to the target. Leave that to professional labs and read their reports (don't trust them blindly;)). When choosing IPS don't try to pen test it (you can create a false sense of security!) but instead watch how many false positives they generate for your network and usability of its user interface and compare all of that to competitive products. Yet another simply way to try IPS quickly without playing with Metasploit (anyway I advise you to learn this tool - pure fun and adrenalin:)) is to send a packet with the same source and destination IP address. Simply from your router or firewall ping it's own outside interface. The IPS should generate 'impossible IP packet' signature. I tried this "technique" with IOS IPS and AIP-SSM. I'm not sure if it works with other products. Of course this test isn't applicable to the standalone IPS sensors, but only to the integrated ones such as AIP-SSM module or IOS IPS. Anyway I don't practice this test by myself because it cannot check other important signature micro engines such as HTTP. I rather use Metasploit and try some most famous working exploits like ms08_067_netapi against unpatched Windows XP SP2 or earlier or for the client side ms10_002_aurora - a legendary IE Aurora bug against unpatched MS Internet Explorer 6 and 7.

If these tests fire signatures then you now that your IPS is working, but it's not "set and forget" kind of device - it requires at least daily event monitoring, hunting potential false and true positives... And don't forget to educate at least one member of your networking crew to handle the IPS because you don't need a spoiled vacation while trying to unblock good traffic that false-positively fired some killer signature;)