freach's blog

Perl: Call by reference and Call by value

Did you know, if you pass an argument to a Perl subroutine, the value in @_ is an alias for the value you passed in? I didn't! This is very useful, because actual this is a "call by reference" and it should be fast, because there is no copying in the subroutine and no setting the value from the subroutine's return.

Example for a call by reference:

sub increment
{ 
    $_[0]++;
}
 
my $foo = 1;
increment($foo); # $foo is now 2

Example for a call by value:

sub increment 
{
    return($_[0]+1);
}
 
my $foo = 1;
$foo = increment($foo); # $foo is now 2

So I benchmarked the following code:

#!/usr/bin/perl
 
use 5.10.0;
use Benchmark;
 
# call by reference
sub call_by_reference
{
    $_[0]+=1;
}
 
# call by value
sub call_by_value
{
    return($_[0]+1);
}
 
timethese(10000000, {
    'call_by_ref' => sub { my $foo = 1; call_by_reference($foo); },
    'call_by_val' => sub { my $foo = 1; $foo = call_by_value($foo); },
});

And YES call by reference is faster, nearly 25% faster.

Results:

Benchmark: timing 10000000 iterations of call_by_ref, call_by_val...
call_by_ref:  7 wallclock secs ( 7.95 usr +  0.00 sys =  7.95 CPU) @ 1257861.64/s (n=10000000)
call_by_val: 10 wallclock secs (10.50 usr +  0.00 sys = 10.50 CPU) @ 952380.95/s (n=10000000)

So, working with references is fast and cheap, but that's nothing new. What I like to know, is there a real difference working directly on @_. I wrote this little test, to prove it.

#!/usr/bin/perl
 
use 5.10.0;
use Benchmark;
 
# call by reference on @_
sub call_by_reference_array
{
    $_[0]++;
}
 
# call by reference with ref copy to scalar
sub call_by_reference_ref
{
    my($ref) = @_;
    $$ref++;
}
 
timethese(10000000, {
    'call_by_ref_array' => sub { my $foo = 1; call_by_reference_array($foo); },
    'call_by_ref_ref' => sub { my $foo = 1; call_by_reference_ref(\$foo); },
});

Results:

Benchmark: timing 10000000 iterations of call_by_ref_array, call_by_ref_ref...
call_by_ref_array:  3 wallclock secs ( 3.10 usr +  0.00 sys =  3.10 CPU) @ 3225806.45/s (n=10000000)
call_by_ref_ref:  3 wallclock secs ( 4.61 usr +  0.00 sys =  4.61 CPU) @ 2169197.40/s (n=10000000)

As you can see, only by copying the passed reference to a scalar in the subroutine, you lose performance. And not only a few operations per second, you lose 1056609.05/s.

Howto recover deleted files from ext2/3

The last weeks I was working on a Perl project and after finishing the job I decided to delete the file by mistake.
First thought .. "You stupid f**k".
So, what to do now? There are two kinds of deleting content.

First one
rm file
You just deleted the file ... good news are, this one is easy to recover.

Install the tool ext3grep an easily recover the file with:
ext3grep --restore-file file device
Example:
You deleted index.pl from /home/www/ and /home is a seperate partition called /dev/sda3.
Command: ext3grep --restore-file home/www/index.pl /dev/sda3

So what ext3grep is doing here is going through all block groups searching for a file named home/www/index.pl and restoring it. This is working for deleted directories, too. Very easy and can be done while the device is mounted.

Second one

echo > file
or 
open(my $fh, '>', 'the_perl_script_im_working_on_because_im_stupid.pl');

This one is a bit more tricky. First you need to get the Sleuthkit (http://www.sleuthkit.org/sleuthkit/download.php). Just download it and built it from source, because the current Debian and Ubuntu packages turned out to be crap, because the tool we need (blkls) is missing. There are no big built dependencies just install build-essential and then ./configure && make .. done. Second tool we need is debugfs, but this one should be installed already, if not install it.

So, you didn't delete the file, it's there, but the size is 0 byte. Again, good news are, content is still available, you just need to handle the file system in "raw mode".

  • You need to find out the inode of the file:
    ls -li file
    ls said your file is on inode 3695750
  • You need to find out to which block group the inode (file) belongs to:
    debugfs /dev/sda3
    debugfs: imap <3695750>
    Inode 3695752 is part of block group 226
    located at block 7405576, offset 0x0380

    Now you know, the file is located at block 7405576 in block group 226
  • Now you need to find out, how big is a block group (usually 32768):
    debugfs: stats
    Have a look at the line saying "Blocks per group", this one is your size of a block group.
  • It's time for the math. You need to find out where the block group starts and where the block group ends.
    Block group start: 226 * 32768
    Block group end: (226 * 32768) + (32768 - 1)
  • Now we can dump this particular block group and extract our file from the raw dump.
    blkfs /dev/sda3 7405576-7438335 > /root/blockgroup-226.raw
  • If everything went fine, you should find your plain file in /root/blockgroup-226.raw and you can extract it from the other parts of the block group.

This stuff can also be done without unmounting the device.
Now, if you don't know the file name you deleted and the file isn't still there as 0 byte file, but you can remember certain lines of your file, like a method name or a variable name you used, ext3grep can be useful.

  • Use ext3grep to search for a string pattern:
    ext3grep --search "sub string_to_hex" /dev/sda3
    Every time ext3grep matches the search string with content of your device, it will print the block.
  • If ext3grep found a block take it and divide it through your blocks per group and round the result down
    7415811 / 32768 = 226.312591553 = 226
    Now you know, the content you are looking for is in block group 226 and you should know what to do now.

Script for auth.log and kern.log analysis

Referring to the articles "GeoIP for iptables on Debian Lenny" and "Connection logging with iptables", I wrote a Perl script, which looks into auth.log and kern.log data to create a little report.

  • auth.log: Is used for analysing connections on your SSH server. It checks for authed and failed connections and summarize them per source IP, with count of authed/failed connections and authed/failed users.
  • kern.log: Is used for analysing any logs from iptables. Any log row will be summarized to blocks by source IP, destination IP, protocol and destination port. The data will be checked on connection count and connection rates, to determine if this source IP is an attacker or not.

Example for auth.log report:

IP: 85.XXX.XXX.XXX (DE)
Connects: 542
Authed connections: 0
Failed connections: 477
Failed users: testing,vincent,test,sales,tt,migrate,mike,user,nagios,portal,admin,apache,usuario,ts,jboss,postgres,cod,susan,murmur,bobcat,testuser,tester,css,basic,www,copier,student,gary,tv,upload,mythtv,ftpuser,michelle,phone,oracle,build,as

Example for kern.log report:

Source IP: 85.XXX.XXX.XXX
Dest. IP: 188.XXX.XXX.XXX
Dest. Port: 22
Source Location: DE
Connection count: 542
Attack: yes
Attack count: 540
Avg connection rate: 0.14 conn/sec
Max connection rate: 0.14 conn/sec
Duration: 77.81 sec

So, I think this output is a bit more handy for humans, than raw log data and you have it at first glance, if there were attacks and how intense.
To execute the script you need a GeoIP database and libgeo-ip-perl installed. Here are some instructions.

aptitude install libtext-csv-xs-perl libgeo-ip-perl
 
mkdir -p /var/geoip/LE /usr/src/GeoIP
wget -O /usr/src/GeoIP/GeoIPCountryCSV.zip http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
wget -O /usr/src/GeoIP/csv2bin-20041103.tar.gz http://people.netfilter.org/peejix/geoip/tools/csv2bin-20041103.tar.gz
wget -O /usr/src/GeoIP/geoip_src.tar.bz2 http://jengelh.medozas.de/files/geoip/geoip_src.tar.bz2
wget -O /usr/local/sbin/intruder.pl.gz http://tuxj0b.de/files/intruder.pl-2010-01-11.gz
 
cd /usr/src/GeoIP
tar xf csv2bin-20041103.tar.gz
tar xf geoip_src.tar.bz2 geoip_csv_iv0.pl
unzip GeoIPCountryCSV.zip
 
cd /usr/src/GeoIP/csv2bin
make
 
cd /var/geoip
/usr/src/csv2bin/csv2bin /usr/src/GeoIP/GeoIPCountryWhois.csv
 
cd /var/geoip/LE
perl /usr/src/GeoIP/geoip_csv_iv0.pl /usr/src/GeoIP/GeoIPCountryWhois.csv
 
gunzip /usr/local/sbin/intruder.pl.gz

The script is developed and tested to be working on Debian Lenny. If there are issues, please let me know!
For better functionality I suggest to set LogLevel DEBUG in /etc/ssh/sshd_config and restart your SSH server. With the default setting you won't be able to track all connections made to your SSH server, only the authed or failed connections. If you like to have daily reports, you should set logrotation for auth.log and kern.log to daily.

/etc/logrotate.d/rsyslog

/var/log/syslog
/var/log/auth.log
/var/log/kern.log
/var/log/messages
{
	rotate 7
	daily
	missingok
	notifempty
	delaycompress
	compress
	postrotate
		invoke-rc.d rsyslog reload > /dev/null
	endscript
}
 
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
{
	rotate 4
	weekly
	missingok
	notifempty
	compress
	delaycompress
	sharedscripts
	postrotate
		invoke-rc.d rsyslog reload > /dev/null
	endscript
}

Default log files for analysis are /var/log/auth.log and /var/log/kern.log. You can pass other files by the directives -authlog PATH and -kernlog PATH.

Usage: intruder.pl [OPTION]
  -a, -authlog Path to auth.log file.
  -k, -kernlog Path to kern.log file.
 
Report bugs to freach at tuxj0b.de

Download: intruder.pl-2010-06-19.gz
MD5SUM: fec6abd1e99f8b40670d398700feb803

Plans for a new mail server tutorial

After receiving a lot of mails regarding the howto on Mailserver mit Postfix, Dovecot, Antispam und PostgreSQL als Backend (Mail server with Postfix, Dovecot, anti spam and PostgreSQL), I've plans for doing a new one.

The basic setup for the new tutorial will be:

  • Debian Lenny
  • Postfix 2.5.5
  • Dovecot 1.2.9
  • MySQL 5.0
  • Virtual domains
  • SMTP auth
  • TLS on SMTP and IMAP
  • Sender verification
  • E-Mail address wildcards

To reach more people, it'll be on English this time. If there are any feature requests from you, just write a comment.

Perl making an array unique

I had to write a Perl tool lately, which can make a list of things unique, like uniq(1) from GNU coreutils. In native Perl, there is no such function, to make an array only having unique values. So I came up with the idea using a hash for making an array unique very fast.

sub array_uniq
{
  my @array = @_;
  my %uniq = ();
 
  for (@array)
  {
    $uniq{$_} = undef; 
  }
 
  return(keys(%uniq));
}
 
my @uniq_array = array_uniq(@not_uniq_array);

I also checked, if there is a difference in performance using map instead of for, but it turned out, in this case, there isn't. After writing the code I had a look at Perl best practices and this way seems to be the most efficient.

Connection logging with iptables

This is a short tutorial on how to log connections for analysis or detecting attacks with iptables. For connection auditing on data critical services like SSH, IMAP, HTTP and so forth, it is handy to log every new connection to these services with things like timestamp and source IP. With such data it's possible to detect a intrusion or maintain present security systems.

This is an example for logging new TCP connections.
For better maintainability we create a new chain for the logging rules and create a new rule in the INPUT chain, which says, all TCP traffic with SYN flag set, jump to LOGGING chain.

iptables -N LOGGING
iptables -A INPUT -p tcp --syn -j LOGGING

Now we add a rule for logging every new TCP connection on port 143 (usual IMAP).

iptables -I LOGGING -p tcp --destination-port 143 -j LOG --log-level info --log-prefix '*LOG*'

iptables is using the kernel log, so we have to specify a log level for syslog. Logging data most of the time is for wasting disk space, but sometimes for further analysis. To make the row grabbing a little easier, we prefix each log entry with *LOG*.
On creation of a new connection on port 143, a row like this will be created.

Jan  6 17:20:49 myhost kernel: [8304902.490996] *LOG*IN=eth0 OUT= MAC=XXX SRC=XXX DST=XXX LEN=60 TOS=0x00 PREC=0x00 TTL=55 ID=5137 DF PROTO=TCP SPT=28889 DPT=143 WINDOW=5840 RES=0x00 SYN URGP=0

In detail, this row will always be created if a TCP packet for destination port 143 got the SYN flag set, so you can detect SYN floods, too.

For detecting other attacks, for example brutforcing on SSH, you can add the following.

iptables -N LOGSSH
iptables -I LOGSSH -j ACCEPT -m limit --limit 5/m
iptables -A LOGSSH -j LOG --log-level info --log-prefix '*ATTACK*'
 
iptables -A LOGGING -p tcp --destination-port 22 -j LOG --log-level info --log-prefix '*LOG*' -m limit --limit 5/m
iptables -A LOGGING -p tcp --destination-port 22 -j LOGSSH

This creates a new chain called LOGSSH, which got 2 rules.
The first rule is using the limit match. The limit match does rate limiting based on packets per time. The first rule means, any TCP packet jumped into LOGSSH chain, will be accepted, while the source IP is in the limit of 5 matches per minute. Once the connecting party is above the limit of 5 connections per minute, the second rule will be triggered, which is logging with prefix *ATTACK*. After that we add 2 additional rules, where the order is very important. First one is for logging all non *ATTACK* connections to port 22 and the second is for jumping into LOGSSH, to check if this is an attack. Log entries will not be duplicated, if it's a non *ATTACK* connection the row will have the prefix *LOG*, if not it will have *ATTACK*, because of the -m limit --limit 5/m in the *LOG* prefixed rule.

These examples are only for analysis or intrusion detection, but you also could use such instruments, to prevent a attack. Please be aware of the data amount, which is logged! It's not a good idea to enable such logging for services, which are known to have a lot of new connections or are a major target for SYN flooding. If you don't care, you will DDoS yourself, with a lot of I/O or even a full hdd.

For further informations on this topic:
- man 8 iptables
- iptables -m limit --help

GeoIP for iptables on Debian Lenny

This is a little tutorial for adding GeoIP support to iptables in Debian Lenny. GeoIP maps real world locations to IP networks. With GeoIP support in iptables, you can restrict access to certian locations.

In my tutorial I will restrict access for countries, which are causing a security threat to hosted services. For short, ban the damn Chinese.

aptitude install libtext-csv-xs-perl linux-headers-`uname -r` iptables-dev
 
mkdir -p /var/geoip/LE /usr/src/GeoIP
wget -O /usr/src/GeoIP/GeoIPCountryCSV.zip http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
wget -O /usr/src/GeoIP/csv2bin-20041103.tar.gz http://people.netfilter.org/peejix/geoip/tools/csv2bin-20041103.tar.gz
wget -O /usr/src/GeoIP/geoip_src.tar.bz2 http://jengelh.medozas.de/files/geoip/geoip_src.tar.bz2
wget -O /usr/src/GeoIP/xtables-addons-1.21.tar.bz2 http://downloads.sourceforge.net/project/xtables-addons/1.21/xtables-addons-1.21.tar.bz2
 
cd /usr/src/GeoIP
tar xf csv2bin-20041103.tar.gz
tar xf geoip_src.tar.bz2 geoip_csv_iv0.pl
unzip GeoIPCountryCSV.zip
tar xf xtables-addons-1.21.tar.bz2
 
cd xtables-addons-1.21
./configure --with-xtlibdir=/lib/xtables
make
make install
 
cd /usr/src/GeoIP/csv2bin
make
 
cd /var/geoip
/usr/src/csv2bin/csv2bin /usr/src/GeoIP/GeoIPCountryWhois.csv
 
cd /var/geoip/LE
perl /usr/src/GeoIP/geoip_csv_iv0.pl /usr/src/GeoIP/GeoIPCountryWhois.csv

Now we can add rules for keeping certian locations out.

iptables -N GEOIP_REJECT
iptables -I GEOIP_REJECT -m geoip --src-cc CN -j REJECT
iptables -I GEOIP_REJECT -m geoip --src-cc KR -j REJECT
iptables -I GEOIP_REJECT -m geoip --src-cc KP -j REJECT
iptables -I GEOIP_REJECT -m geoip --src-cc TW -j REJECT
 
iptables -A INPUT -j GEOIP_REJECT

Drupal Geshifilter Plugin sorgt für Segfault bei FastCGI Instanzen

Aus einem unerklärlichen Grund, ist wohl seit kurzer Zeit, das Syntax-Highlighting-Plugin für mein Drupal Blog, auf die Idee gekommen ein Segfault bei den FastCGI Instanzen meines LightTPDs auszulösen. Das ganze äußerte sich dann in einem "500 - Internal Server Error". Mein Dank geht an einen aufmerksamen Leser, der mich freundlicher Weise auf das Problem aufmerksam gemacht hat!! Fix, Plugin und Geshi Lib update.

Falls wer ähnliche Probleme hat, hier der strace Auszug:

fstat64(3, {st_mode=S_IFREG|0640, st_size=113265, ...}) = 0
_llseek(3, 0, [0], SEEK_CUR)            = 0
read(3, "<?php\n/**\n * GeSHi - Generic Syn"..., 8192) = 8192
read(3, " string\n     */\n    var $footer_"..., 8192) = 8192
read(3, "t declarations\n     *\n     * @pa"..., 8192) = 8192
read(3, "recated In favour of set_symbols"..., 8192) = 8192
mmap2(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb69fa000
read(3, "    * method without parameters "..., 8192) = 8192
read(3, "he url somewhere, it is replaced"..., 8192) = 8192
read(3, "de, $i - strlen($close) + 1, str"..., 8192) = 8192
read(3, "SCAPE\'] as $hard) {\n            "..., 8192) = 8192
read(3, "    else {\n                     "..., 8192) = 8192
read(3, " Fix to reduce the number of rep"..., 8192) = 8192
read(3, "   //\n        foreach ($this->la"..., 8192) = 8192
read(3, "IDs for line numbers, there need"..., 8192) = 8192
mmap2(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb69b9000
read(3, "\n        if ($this->header_type "..., 8192) = 8192
read(3, "Additionally, set default styles"..., 8192) = 6769
read(3, "", 8192)                       = 0
read(3, "", 8192)                       = 0
close(3)                                = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---

LightTPD unter Debian 4.0 mit Lua-Support kompilieren

In der neusten Version von LightTPD wird für mod_magnet und mod_cml Lua ab Version 5.1 benötigt. Damit das Kompilieren mit Lua-Support funktioniert muss das configure-Script von LightTPD editiert werden, da sonst die installierte Lua-Version nicht erkannt wird.

Lua-Support installieren:
# aptitude install lua5.1 liblua5.1-0 liblua5.1-0-dev

Im configure-Script von LightTPD auf Zeile 26328 WITH_LUA=lua nach WITH_LUA=lua5.1 ändern.
Danach könnt ihr eueren LightTPD mit Lua-Support wie gewohnt kompilieren.

IPSec Site-to-Site: Openswan Einwahl auf Draytek Vigor Router

Es war eine mehr als schwere Geburt ein Site-to-Site VPN mit Openswan als Client und einem Draytek Vigor Router als Server herzustellen. Hier nun die Auflösung wie simple es doch sein könnte.

1. Netzwerktopologie
left = Openswan
right = Draytek Vigor Router
10.0.1.0/24 ... 194.1.2.3 == 194.3.2.1 ... 10.0.2.0/24

Es wird ein Site-to-Site VPN für das Subnetz 10.0.1.0/24 und 10.0.2.0/24 über die öffentlichen IPs der Gateway hergstellt. Bei der Verbindung wird kein NAT verwendet! Für die Authentifizierung wird PSK genutzt und die Verbindung mittels ESP im Tunnelmodus aufgebaut.

2. Installation Openswan
# aptitude install openswan

3. Konfiguration Openswan
/etc/ipsec.conf:

version	2.0
 
config setup
	plutodebug=none
	klipsdebug=none
	myid=194.1.2.3
 
conn site-to-site
	type=tunnel
	connaddrfamily=ipv4
	left=194.1.2.3
	leftsubnet=10.0.1.0/24
	right=194.3.2.1
	rightsubnet=10.0.2.0/24
	keyexchange=ike
	auto=add
	auth=esp
	authby=secret
	dpddelay=10
	dpdtimeout=300
	pfs=no
	keylife=28800
	rekey=no
	keyingtries=2
	ikelifetime=3600
	compress=no
 
#Disable Opportunistic Encryption
include /etc/ipsec.d/examples/no_oe.conf

/etc/ipsec.secrets:
194.1.2.3 194.3.2.1 : PSK "dasgeheimepasswort"

3. Konfiguration Vigor Router
VPN und externe Einwahl > LAN-zu-LAN: Neues Profil anlegen
- 3. Allgemeine Einstellungen:
- Einwahl zulassen über IPSec
- Pre-shared Key
- IPSec Sicherheitsmethode: Hoch ESP (AES)

- 4. TCP/IP Netzwerk-Einstellungen:
- Meine WAN-IP: 194.3.2.1
- Remote Gateway-IP: 194.1.2.3
- Remote Netzwerk-IP: 10.0.1.0
- Remote Netzwerk-Maske: 255.255.255.0

4. Starten der IPSec-Verbindung
Aus irgendeinem Grund konnte bei meinen Tests nie eine Verbindung automatisch aufgebaut werden (auto=start), daher initiere ich die Einwahl manuell.
# ipsec auto --up site-to-site

Nun wird der Tunnel aufgebaut und eure lokalen Subnetze können über den IPSec-Tunnel kommunizieren. Um den Tunnel zu Testen versucht aus dem Netz 10.0.1.0 einen Host aus dem Netz 10.0.2.0 zu erreichen. Bedenkt, dass durch eine IPSec-Restriktion vom Gateway kein Host aus dem jeweils anderen Subnetz erreicht werden kann!!

Syndicate content