One of the most procrastinated issues I had at a Customer’s, was the proliferation of errors like these (as shown in servers/clients Event Viewer):

Event Type: Error
Event Source:   crypt32
Event Category: None
Event ID:   8
Description:
Failed auto update retrieval of third-party root list sequence number from: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootseq.txt> with error: This network connection does not exist.
Event Type: Error
Event Source:   crypt32
Event Category: None
Event ID:   11
Description:
Failed extract of third-party root list from auto update cab at: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab> with error: A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.
There are several posts mentioning the issue, this one pointed me in the right direction. Basically, because of how SEP components communicate, Windows is triggered into updating the list of trusted root Certification Authorities. It tries to do so through the Internet using the Computer account. The latter may not have any proxy configured. Being unable to reach outside, the host gets flooded by crypt32 errors.

In order to solve the issue, I decided to deploy a valid proxy configuration, for the Computer account (SYSTEM user), on a subset of the Domain’s hosts.
One of the ways to script that is the “proxycfg -u” command1 that works by copying the current user proxy settings to the SYSTEM’s registry. Sounds cool but if the current user is not a member of the local Administrators group, he won’t have the necessary rights. The following script instead, can be launched via Group Policy2 during operating system startup, and since it’s a startup script rather than a login one, it will run with administrative privileges.

Nothing fancy in the below source. It creates the registry key if it doesn’t exist, then sets the right value for WinHttpSettings which I obtained this way:

use “proxycfg -u” on a test host
use the Registry editor to export the contents of HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections

The value is of type REG_BINARY. Since the RegWrite API (method of class WScript.Shell) cannot deal with binary values, WMI (StdRegProv registry provider) needs to be used. Also, SetBinaryValue expects an array of decimal values, while Regedit exports them as hexadecimal digits (you’ll have to take care of the conversion yourself).

On Error Resume Next
Const HKEY_LOCAL_MACHINE = &H80000002

strPath = "SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections"
strKey = "WinHttpSettings"
strValue = "24,0,0,0,0,0,0,0,3,0,0,0,19,0,0,0,112,114,111,120,121,46,99,117,115,116,46,108,97,110,58,56,48,56,48,47,0,0,0,49,48,46,42,46,42,46,42,59,115,101,114,118,101,114,50,48,59,115,101,114,118,101,114,50,48,46,42,59,42,46,99,117,115,116,46,108,97,110,59,60,108,111,99,97,108,62"
strMachineName = "."

arrValues = Split(strValue,",")
strMoniker = "winMgmts:\\" & strMachineName & "\root\default:StdRegProv"
Set oReg = GetObject(strMoniker)
rv = oReg.CreateKey(HKEY_LOCAL_MACHINE, strPath)
rv = oReg.SetBinaryValue(HKEY_LOCAL_MACHINE, strPath, strKey, arrValues)
If the scripts works as it should, you’ll be greeted by these events:

Event Type: Information
Event Source:   crypt32
Event Category: None
Event ID:   7
Description:
Successful auto update retrieval of third-party root list sequence number from: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootseq.txt>
Event Type: Information
Event Source:   crypt32
Event Category: None
Event ID:   2
Description:
Successful auto update retrieval of third-party root list cab from: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab>
And, hopefully, crypt32 errors will be gone for good.

See Using the WinHTTP Proxy Configuration Utility ↩
Computer Configuration, Windows Settings, Scripts, Startup ↩

A newly installed FortiGate cluster (a simple two node HA active-passive setup) and some packet loss issues…
Ping from the LAN side to the Internet (or from the firewall itself) resulted in about 20% packet loss, while the other way around (WAN to firewall’s main public IP) didn’t work at all.

I used the following command to check my MAC addresses:

FORTIGATE-PRI # diagnose hardware deviceinfo nic wan1
[..]
Current_HWaddr                  00:09:0f:09:00:08
Permanent_HWaddr                00:09:0f:d1:be:ef
[..]
Then resorted to the “show mac” switches facilites (some Cisco, some ProCurve) to know on which network ports that particular MAC lied… Only to discover that the cluster’s “logical” MAC address (00:09:0f:09:00:08) wasn’t really located where I expected it to be.
Well, FortiGate’s MAC addresses aren’t randomly generated. They have predictable values that depend on the firewall’s port number. The eight port (or wan1, in my case) will always have a virtual MAC as the one above. What will happen if you have two clusters (as we had) sitting on the same L2 network segment (on the same broadcast domain, that is)? You said MAC address conflict? You’re right.
The solution is simple, use the group-id directive to tweak the logical MAC address, i.e.:

config system ha
    set group-id 10
end
Changes the second right-most bytes of the MAC, from 00 to 0a:

before  00:09:0f:09:00:08
after   00:09:0f:09:0a:08
Point is that the “FortiOS High Availablity Handbook” explains the case very thoroughly! See page 192, paragraph “Diagnosing packet loss with two FortiGate HA clusters in the same broadcast domain”. We’re so used to discardable product documentation that sometimes we don’t even try to look for clues where they should normally reside.
Instead of troubleshooting, this time, I should really have Read The (unexpectedly) Fine Manual…

Today Internet browsing is particularly slow.
At seemingly random intervals, available bandwith drops down and people get more and more irritable.

How do you find out why this is happening?

The possible causes boil down to:

Router/Firewall1 is not pleased by “something”. Could be an attack or a bug in the device firmware.
Too many connections. Maybe they’re not passing much traffic, but the internet gateway can’t keep up with their number. I’ve seen firewalls perform very badly in this respect. E.g.: 3 connections trying to download/upload as fast as they can, and a total, aggregate, b/w of 10Mbps. Those 3 plus 3000 “normal” connections and a total b/w of 6Mbps.
A reasonable amount of connections, effectively eating all of the available bandwidth.

I’ll skip case A, for now.
In case B you’ll likely want to know the firewall’s idea of “netstat”, meaning the complete listing of TCP/UDP/other connections. No big deal if the device has got some sort of CLI access: capture its output, import it into a spreadsheet, or use awk/sort/grep2 to build your stats. Usually, computing total number of connections by source IP address and sorting accordingly, is enough to gain some insight about what’s going on.
Case C… For long-running (days) data analysis, you could use a tool like NTOP. But if, like me today, you need to act quickly (perhaps because you know that the issue will disappear soon), iftop can hardly be beaten.
Both tools require the machine they run on to be able to “sniff” all the traffic passing through the firewall. This can be accomplished by configuring monitoring/monitored port(s) on a switch. Monitored ports get their inbound/outbound traffic copied to the monitoring one. Different vendors call the thing a different way, port mirroring is also a good keyphrase. Here are a couple of resources:

(Old) 3Com Superstack: Monitor Port on 3Com 4400
HP ProCurve, pretty straightforward to set up using the “menu” interface: How do I attach a LAN Analyzer to a Switch 208t/224t port to monitor LAN traffic for diagnostic purposes?
ProCurve switches are not limited to mirroring ports that belong to the same device/chassis: How to configure remote and intelligent mirroring on ProCurve switches

Low-end HP switches (like the ProCurve 1800 one I encountered here), though, are only manageable via a web gui:
Port Mirroring on a ProCurve 1800

Cisco: Port Mirroring, Configuring a Cisco Catalyst Switch SPAN mirroring port

(You could as well use a hub instead of a switch and get implicit mirroring of any port, to any port of the hub. Just unplug the firewall, link the hub to the switch, plug firewall and monitoring host in the hub. Kludgy but quick and easy, if you can afford the temporary cabling changes, and the bottleneck introduced by the hub…)

So:

Find the switch where the firewall is connected to. Which side of the firewall? It depends on where you believe the issues originates from. Let’s say the culprit is most likely to lie on the LAN → switch port A.
Connect your laptop/monitoring machine to the same switch → port B.
Set up monitoring: port A is monitored, port B is monitoring.
Run iftop, maybe telling it to also show port numbers (“-P”, without this switch, you’ll only see totals by source/destination IP addresses couple), don’t display hostnames “-n”, the interface “-i eth0″ and provide a meaningful filter (here I’m selecting packets whose source is not on the LAN3. The “-p” option instructs iftop to capture packets in promiscuous mode. Without it, iftop won’t lift off the wire packets that aren’t addressed to the machine on which it is running.
iftop -p -P -n -i eth0 -f 'not src net 192.168.200.0/23'
Iftop will produce a realtime table of running connections, sorted by how demanding they are in terms of bandwidth (10s average, by default). See the screenshot below; the top connections are due to two running video conference streams stealing 1Mbit/second worth of bandwidth, each.
iftop's output

Once everything is set up and you’re able to read iftop’s output, spotting the “top talkers” of your net becomes kids play, enjoy!

for brevity, I’ll just say “firewall” from now on. ↩
Yuri is king at doing that. See his AWK weekly series. ↩
iftop will still show these source addresses, since its output is always made of bidirectional “connections”. Only, counters pertaining to the LAN → outside direction, won’t increase. ↩

This post1 explains how to use nmap and smb-check-vulns to scan a network in search of Conficker infected hosts. I thought that the whole Conficker case was over, but hopefully some of the measures I took to deal with it almost an year ago, will still be relevant to other kinds of malware. And, also, the method I’ll show you here differs from the nmap one in that the latter is active, whereas mine is passive. Actively probing an host for vulnerabilities could be very very much alike “exploiting” it as malware does, and have similar effects. For instance, a service/process could crash, making it not always advisable to run active scans on your servers subnet. Passive analysis, on the other hand, unobtrusively collects clues about who’s misbehaving.

During the Conficker/Downadup outburst, we observed that:

Antivirus wasn’t always able to detect/stop it.
The virus was copying files in known directories (C:\WINDOWS\SYSTEM32) on about to be infected machines.
Security patched hosts were still subject to the remote malicious file copying routine. The copy could either succeed or fail, depending on which permissions had the user that “runs” the virus. The copy in itself doesn’t pose any security concern. Even if no A/V is active on the destination host, but virus exploitable flaws have been patched, malware won’t be able to activate itself. Otherwise, the A/V would remove suspect files as soon as they are caught, without interfering with our detection purposes.

This behaviour makes it possible to use a “honeypot” approach. The detecting server can be any production host provided that it is security patched and A/V protected. You could, as we did, choose a Domain Controller and:

Run Administrative Tools → Domain Controller Security Policy

Modify the Audit Policy, enabling tracking of successful logon events and object access. By default the OS will only log failures, but that’s not enough.

Object Access is activated at a file/directory level. Open up the Properties of a directory you know is accessed by the virus, click on Security, then Advanced. The Auditing tab is what you’re interested in. Set things up so that any “Create File/Write Data” attempt of Type “Success” will be logged. The semantics about how auditing settings are propagated from parent to child works in the same way as NFTS permissions.

From this point on, you should monitor the honeypot server’s Security Event Log. I wrote a Perl script to do it for me. It works by selecting events with ID 560 and 540, extracting their text and printing just the needed info.

Let’s look at how it’s used (the only parameter is the hostname/address of the honeypot server):

C:\loganalysis>perl ddloganalysis.pl honeypot-srv.domain.lan > ddlog.txt
Skimming through the generated log, you’ll notice the files being dropped into C:\WINDOWS\system32 (or any directory you set up for auditing), the user that actually created them and, before (time-wise), from which address the user is coming.

17/03/2009 16.26.19   560 : C:\WINDOWS\system32\onevthx.vr (Administrator)
17/03/2009 16.26.18   540 :  (10.1.1.94 - Administrator)
17/03/2009 15.35.24   560 : C:\WINDOWS\system32\onevthx.vr (SpectrumLT)
17/03/2009 15.35.24   540 :  (10.6.3.6 - SpectrumLT)
We successfully used the script to pinpoint the rogue hosts. Deeming it useful, here it is:

#!perl

use strict;
use Win32::EventLog;
use POSIX qw ( strftime );

my @matches = (
    #'job$',   # useless, since scheduled tasks are always created by SYSTEM
    'system32',
    'eicar.com'
);

die "Usage:\n$0 servername" unless $ARGV[0];

my $ev=Win32::EventLog->new('Security', $ARGV[0])
    or die "Can't open EventLog\n";
my $recs;
$ev->GetNumber($recs)
    or die "Can't get number of EventLog records\n";
my $base;
$ev->GetOldest($base)
    or die "Can't get number of oldest EventLog record\n";

sub getts($) {
    return strftime '%d/%m/%Y %H.%M.%S', (localtime shift);
}

my @progress = ('-','\','|','/','-','\','|','/');

my $x = $recs-1;
my $h;
while ($x >= 0) {
    $ev->Read(EVENTLOG_FORWARDS_READ|EVENTLOG_SEEK_READ,
        $base + $x,
        $h)
        or die "Can't read EventLog entry #$x\n";
    print STDERR $progress[$#progress - ($x % @progress)] . "\r";
    if ($h->{Source} eq 'Security' and ($h->{EventID} == 560 or $h->{EventID} == 540)) {
        Win32::EventLog::GetMessageText($h);
        if ($h->{EventID} == 560) {
            $h->{Message} =~ /Object Name:[\t ]*(.*?)\r/gis;
            my $filename = $1;
            $h->{Message} =~ /Client User Name:[\t ]*(.*?)\r/gis;
            my $clientusername = $1;
            if ($filename) {
                if (grep { my $m = $_; $filename =~ /$m/i} @matches) {
                    printf "%s %5d : %s (%s)\n", getts($h->{TimeGenerated}), $h->{EventID}, $filename, $clientusername;
                }
            }
        } elsif ($h->{EventID} == 540) {
            $h->{Message} =~ /User Name:[\t ]*(.*?)\r/gis;
            my $username = $1;
            $h->{Message} =~ /Workstation Name:[\t ]*(.*?)\r/gis;
            my $workstation = $1;
            $h->{Message} =~ /Source Network Address:[\t ]*(.*?)\r/gis;
            my $addr = $1;
            printf "%s %5d : %s (%s - %s)\n", getts($h->{TimeGenerated}), $h->{EventID}, $workstation, $addr, $username
              if $workstation or $addr;
        }
    }
    $x--;
}

exit;

In italian, sorry. Look here for an english equivalent and here for more info. ↩

The other day I swapped a firewall with a different one, a FortiGate 60B. After having re-created the config, everything seemed to be functional but: Internet browsing “felt” a bit sluggish (I was on a 20Mbps uplink) and, here comes the weirdness, when I did “something” the whole WAN connectivity would just hang for a couple of minutes. The issue was reproducible by trying to connect via Remote Desktop to one of the published servers (by tunneling through my Employer’s Office, and bouncing back on the Customer’s firewall) or even by opening my Flickr page (but then the cause could’ve been the poor quality of the pictures therein ).
At first, I thought about a dreadful MTU issue, maybe the firewall/router or something along the road was choking when fragmenting or reassembling packets. But, a “ping outside_host -s 1472 -M do” (or “ping -f -l 1472 outside_host“, on Windows) proved that ICMP packets 1500 bytes big (1472 bytes of payload, plus 28 bytes of ICMP header) could indeed flow out and back without being fragmented: the issue was totally random.
Besides that, even lowering the MTU on my PC wouldn’t change anything.
After much cursing, I tried to see if anything was going on at L2 level. Firewall and router (Cisco, owned by the ISP, not accessible to me) were connected together by a crossover cable.
The relevant FortiOS CLI command is the following:

FIREWALLNAME # diagnose hardware deviceinfo nic wan1
System_Device_Name              wan1
Link                            up
Speed                           100 Mbps full duplex
FlowControl                     Tx off, Rxoff
MTU_Size                        1500
My firewall (the above example comes out from another one) was negotiating 100Mbps speed, Half Duplex. Nothing wrong with that, I tried to fix these parameters on the FortiGate but the Ethernet link would not come up. So, auto-negotiation was mandatory and I had no way to change that on the router.
At some point, when Internet connectivity was stuck, it seemed to me that unplugging and plugging back in the cable between firewall/router, would allow for a faster recovery. Definitely, something was wrong at L2.
The solution was to insert a 15€ DLink switch between firewall and router. No problems since then, it really looks like FortiGate and Cisco NICs don’t play well together, at least in that conditions. The Customer will call the ISP in order to tweak the settings Cisco side and see if they can get rid of the switch.
The proper way to diagnose the problem would’ve been to ping the router from the outside during a connectivity stop. Since the issue was “local”, the router should answer while no traffic should pass from the firewall to the router.

Like
Unlike