Sunday, November 20, 2016

ruby vs bash benchmark (loops comparison).

Bash relies highly on external commands which make it having a lot of overheads; it's the language of the command line, not a proper programming language; it'll work well when most of time is spend by external binaries.

time ruby bench.rb > /dev/null; time bash bench.sh > /dev/null

real    0m0.875s
user    0m0.869s
sys     0m0.007s

real    0m10.336s
user    0m10.111s
sys     0m0.223s

There's no comparison. ruby is magnitudes faster than bash

The scripts --

Bash --

#! /bin/bash
declare -i i
i=0
while test $i -le 999999
do
    echo hello world
    i=i+1
done

Ruby --

#! /bin/ruby
i = 0
while (i <= 999999)
    puts "hello world"
    i = i + 1
end

In the bash binary there's frequent execution of 2 independent binaries -- test and echo which makes it slow.

Thursday, November 17, 2016

nginx+fail2ban tutorial/document.

fail2ban + Nginx

In this system fail2ban is supposed to parse nginx logs (customized) for 404 and 403 status codes and add iptables rules to block IPs on the network layer from which excessive 404 and 403 are coming up.

Under a DDOS, because of the verity of IPs available, the frequency of banning and unbanning will be large, as a result there the iptables command will run too many times, resulting in an overhead. A system has been created to prevent this overhead even when there are 1000s of Ips being banned and unbanned.

Objective is to prevent overload of the application, brute force attacks by sending frequent failed authentication requests. 404s have also been taken care of to prevent path discovery apart from the same reasons as previously stated.

Architecture

Instead of the banning iptables being run directly by fail2ban, it's indirectly executed by a bash script on a cron job which runs a single iptables command to ban/unban any no. of IPs in bulk.

fail2ban runs as an unprivileged user, writes to files containing the IPs to be banned/unbanned which the script parses and bans/unbans them in bulk using a single execution of iptables command.

Implementation

Since this is done for testing purposes on a minimal local system (Gentoo) which runs a custom kernel (no iptables FILTER table support), a Debian VM will be created which will contain the actual implementation of the project.

Hits to the VM will be done from the base machine.

Prepare VM --

$ cat /etc/gentoo-release

Gentoo Base System release 2.3

Create rootfs image from template --

qemu-img create -f qcow2 -o cluster_size=512,lazy_refcounts=on,backing_file=Debian8NetworkedSSHRepoPackagesEnhancedUpdate.qcow Debian8NetworkedSSHRepoPackagesEnhancedUpdate_fail2ban.qcow 20G

Load KVM modules (not loaded because of minimum and highly customized OS) --

modprobe kvm_intel

Create tap device veth for the VM to connect to the base machine --

modprobe tun;ip tuntap add mode tap veth

Assign ipv6 and ipv4 addresses on a temporary basis --

ip a add fc00::1:1/112 dev veth;ip link set dev veth up

ip a add 192.168.3.1/24 dev veth

Enable KSM --

echo 1 > /sys/kernel/mm/ksm/run

echo 30000 > /sys/kernel/mm/ksm/sleep_millisecs

Start VM –

qemu-system-x86_64 -machine accel=kvm,kernel_irqchip=on,mem-merge=on -drive file=/home/de/large/VM_images/Debian8NetworkedSSHRepoPackagesEnhancedUpdate_fail2ban.qcow,id=centos,if=ide,media=disk,cache=unsafe,aio=threads,index=0 -vnc :1 -device e1000,id=ethnet,vlan=0 -net tap,ifname=veth,script=no,downscript=no,vlan=0 -m 512 -smp 4 -daemonize -device e1000,id=inet,vlan=1,mac=52:54:0F:12:34:57 -net user,id=internet,net=192.168.2.0/24,vlan=1

Login to the VM –

$ ssh root@fc00::1:2

root@fc00::1:2's password:



The programs included with the Debian GNU/Linux system are free software;

the exact distribution terms for each program are described in the

individual files in /usr/share/doc/*/copyright.



Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent

permitted by applicable law.

Last login: Thu Nov 17 12:03:07 2016 from fc00::1:1

root@LINUXADMIN:~#

Configure ipv4 address for the VM

In /etc/network/interfaces –

source /etc/network/interfaces.d/*



# The loopback network interface

auto lo

iface lo inet loopback



# The primary network interface

allow-hotplug eth0

iface eth0 inet6 static

address fc00::1:2

netmask 112

#gateway fc00::1

# dns-* options are implemented by the resolvconf package, if installed

#dns-nameservers fc00::1

#dns-search LinuxAdmin



iface eth0 inet static

address 192.168.3.2

netmask 24



auto eth1

iface eth1 inet dhcp

Bring up the changes via console –

ifdown eth0; ifup eth0

Setup nginx –

This setup is just for testing.

aptitude install nginx

The following NEW packages will be installed:

fontconfig-config{a} fonts-dejavu-core{a} geoip-database{a} libfontconfig1{a} libgd3{a} libgeoip1{a} libjbig0{a}

libjpeg62-turbo{a} libtiff5{a} libvpx1{a} libxml2{a} libxpm4{a} libxslt1.1{a} nginx nginx-common{a} nginx-full{a}

sgml-base{a} xml-core{a}

0 packages upgraded, 18 newly installed, 0 to remove and 27 not upgraded.

Need to get 6,076 kB of archives. After unpacking 16.7 MB will be used.

Do you want to continue? [Y/n/?]



systemctl enable nginx

Synchronizing state for nginx.service with sysvinit using update-rc.d...

Executing /usr/sbin/update-rc.d nginx defaults

Executing /usr/sbin/update-rc.d nginx enable

root@LINUXADMIN:~# systemctl start nginx

Setup virtualhost –

rm /etc/nginx/sites-enabled/default

Create /etc/nginx/conf.d/default.conf

server {

listen *:8080;

root /home/docroot;

}

Setup custom log format for nginx as per requirement, tune it as per VM specs –

user www-data;

worker_processes 1;

pid /run/nginx.pid;



events {

worker_connections 768;

# multi_accept on;

}



http {



##

# Basic Settings

##



sendfile on;

tcp_nopush on;

tcp_nodelay on;

keepalive_timeout 65;

types_hash_max_size 2048;

# server_tokens off;



# server_names_hash_bucket_size 64;

# server_name_in_redirect off;



include /etc/nginx/mime.types;

default_type application/octet-stream;



##

# Logging Settings

##



log_format custom "[$time_local] $remote_addr $status $request";

access_log /var/log/nginx/access.log custom;

error_log /var/log/nginx/error.log;



##

# Gzip Settings

##



gzip on;

gzip_disable "msie6";



# gzip_vary on;

# gzip_proxied any;

# gzip_comp_level 6;

# gzip_buffers 16 8k;

# gzip_http_version 1.1;

# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;



##

# Virtual Host Configs

##



include /etc/nginx/conf.d/*.conf;

include /etc/nginx/sites-enabled/*;

}

Test nginx and start –

nginx -t

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok

nginx: configuration file /etc/nginx/nginx.conf test is successful

systemctl start nginx

Setup fail2ban –

aptitude install fail2ban

The following NEW packages will be installed:

fail2ban file{a} libmagic1{a} libpython-stdlib{a} libpython2.7-minimal{a} libpython2.7-stdlib{a} mime-support{a}

python{a} python-minimal{a} python-pyinotify{a} python2.7{a} python2.7-minimal{a} whois{a}

0 packages upgraded, 13 newly installed, 0 to remove and 0 not upgraded.

Need to get 4,687 kB of archives. After unpacking 20.6 MB will be used.

Do you want to continue? [Y/n/?]

Configure fail2ban to start as unprivileged user –

mkdir /var/fail2ban

useradd -G adm fail2ban

chown fail2ban /var/fail2ban

Group adm is to allow fail2ban to read nginx access logs.

Allow fail2ban user to write logs –

chown fail2ban /var/log/fail2ban.log

Modify fail2ban logrotation config to create new empty log files with the correct permission –

/var/log/fail2ban.log {



weekly

rotate 4

compress



delaycompress

missingok

postrotate

fail2ban-client flushlogs 1>/dev/null

endscript



# If fail2ban runs as non-root it still needs to have write access

# to logfiles.

# create 640 fail2ban adm

create 640 fail2ban adm

}

Create /etc/fail2ban/fail2ban.local to make changes to allow running as the unprivileged user –

[Definition]

socket = /var/fail2ban/fail2ban.sock

pidfile = /var/fail2ban/fail2ban.pid

Make changes to /etc/default/fail2ban –

FAIL2BAN_USER="fail2ban"

Start and enable fail2ban –

systemctl start fail2ban

systemctl enable fail2ban

Synchronizing state for fail2ban.service with sysvinit using update-rc.d...

Executing /usr/sbin/update-rc.d fail2ban defaults

Executing /usr/sbin/update-rc.d fail2ban enable

Create actions –

cat /etc/fail2ban/action.d/nginx.local

[Definition]

actionban = echo -n <ip>, >> /var/fail2ban/ban

actionunban = echo -n <ip>, >> /var/fail2ban/unban

As stated before, these actions append to a file containing the IPs to be banned/unbanned as CSV values (that's why >> has been used).

Create filters –

cat /etc/fail2ban/filter.d/nginx40{3,4}.local

[Definition]

failregex = ^\[ \+0530\] <HOST> 403 .*$

[Definition]

failregex = ^\[ \+0530\] <HOST> 404 .*$

The anchors (^, $) specify that the whole log has been considered.

Create the jail –

cat /etc/fail2ban/jail.local

[nginx_403]

filter = nginx403

logpath = /var/log/nginx/access.log

action = nginx

findtime = 30

maxretry = 5

bantime = 300

usedns = no

enabled = true



[nginx_404]

filter = nginx404

logpath = /var/log/nginx/access.log

action = nginx

findtime = 30

maxretry = 50

bantime = 120

usedns = no

enabled = true



[ssh]

enabled = false

Since ssh service was not a part of the project, but enabled in fail2ban by default on Debian, it has been disabled here.

Make fail2ban read the changes and verify status of jails –

fail2ban-client reload

fail2ban-client status

Status

|- Number of jail: 2

`- Jail list: nginx_404, nginx_403

Create iptables scripts to read files /var/fail2ban/ban, /var/fail2ban/unban and add iptables rules.

cat /usr/bin/fail2ban_iptables.sh

#! /bin/bash

PATH="$PATH:/sbin"

if test -e /var/fail2ban/ban

then

iptables -A INPUT -s `cat /var/fail2ban/ban | sed s/,$//` -j DROP

rm /var/fail2ban/ban

fi



if test -e /var/fail2ban/unban

then

iptables -D INPUT -s `cat /var/fail2ban/unban | sed s/,$//` -j DROP

rm /var/fail2ban/unban

fi

Changes to PATH environment variables are there since cron has a very minimal set of executable search paths.

Fix permissions of the file –

chmod 744 /usr/bin/fail2ban_iptables.sh

Make a cron job to execute the script as root –

root@LINUXADMIN:~# crontab -l | grep -v ^\#

* * * * * /usr/bin/fail2ban_iptables.sh

Testing –

2016-11-17 12:54:57,403 fail2ban.actions[1500]: WARNING [nginx_404] Ban 192.168.3.1

cat /var/fail2ban/ban

192.168.3.1,

After some time (once cron job runs) –

iptables -L

Chain INPUT (policy ACCEPT)

target prot opt source destination

DROP all -- 192.168.3.1 anywhere



Chain FORWARD (policy ACCEPT)

target prot opt source destination



Chain OUTPUT (policy ACCEPT)

target prot opt source destination

The same client on hitting the server –

wget --timeout 5 http://192.168.3.2/xyzz

--2016-11-17 12:55:04-- http://192.168.3.2/xyzz

Connecting to 192.168.3.2:80... failed: Connection timed out.

Retrying.



--2016-11-17 12:55:10-- (try: 2) http://192.168.3.2/xyzz

Connecting to 192.168.3.2:80... failed: Connection timed out.

Retrying.



--2016-11-17 12:55:15-- (try: 3) http://192.168.3.2/xyzz

Connecting to 192.168.3.2:80... failed: Connection timed out.

Retrying.



--2016-11-17 12:55:22-- (try: 4) http://192.168.3.2/xyzz

Connecting to 192.168.3.2:80... failed: Connection timed out.

Retrying.

After 2 minutes –

2016-11-17 12:56:57,541 fail2ban.actions[1500]: WARNING [nginx_404] Unban 192.168.3.1

cat /var/fail2ban/unban

192.168.3.1,

After some time (once cron job runs) –

Chain INPUT (policy ACCEPT)

target prot opt source destination



Chain FORWARD (policy ACCEPT)

target prot opt source destination



Chain OUTPUT (policy ACCEPT)

target prot opt source destination



Sunday, November 13, 2016

HTTRACK 'pasues' from time to time when mirroring.

Tired of this behaviour? No solution on the Internet? Mirroring progress slows down?

Try switch -C0 or -C1. With -C0, you wont be able to continue from where you left off last and with -C1, you wont be able to use the cache for fast updates.
So -C1 is better.

Saturday, August 20, 2016

Linux veth device benchmark with high and low mtu.

Veth devices where created --

ip link add name tstveth mtu 65535 type veth peer name tstveth0 mtu 65535

65535 is the highest MTU these devices support.

They where moved to different namespaces --

ip netns add transfertest

ip link set netns transfertest dev tstveth0

IPs where added to it --

ip netns exec transfertest ip a add fc00::2:2/112 dev tstveth0
ip a add fc00::2:1/112 dev tstveth

sshd was listening on fc00::2:1 with compression disabled.

This command was run to test throughoutput of ssh and the sys CPU utilization --

ip netns exec transfertest ssh -i /etc/mypc/my.key -p 80 de@fc00::2:1 dd bs=10M if=/dev/zero > /dev/null
ip netns exec transfertest ssh -i /etc/mypc/my.key -p 80 de@fc00::2:1 dd bs=10M count=103 if=/dev/zero > /dev/null

For the 65535 MTU, CPU was between 6 and 7, sometimes 5 also. Throughoutput was around 140MB/s

When the MTU was lowered to 1500, the CPU utilization dropped to between 5 and 6%, sometimes goes to 7% and the throughoutput was a little higher.

I blame it on chance, but overall, it doesn't make a difference.

Wednesday, April 13, 2016

Enable electrolsys/e10s/multiprocessing on Firefox 45/ESR

Open "about:config" in the URL bar.

Search for "browser.tabs.remote.autostart" and set it to true, then restart.

That's it!

If it's really enabled, in about:support you must see Multiprocess Windows set to true.

Tuesday, March 29, 2016

Apache vs Nginx benchmark/How to make Apache faster than Nginx.

Unlike other benchmarks, both Apache and and Nginx have been tuned for best performance for the task they're doing (serving static content).
Test using apache's ab utility.

Results --

Direct hits --

















Rewrite hits --



















Test done

There are 2 sets of tests done --
  • Hitting URLs with rewrite rules
  • Hitting URLs with file paths directly.
For each of these, test where done from concurrency 1000 to 20000.
In the charts, legends which represent tests done by hitting URLs with file paths directly is suffixed with _direct, while for the rewrite rules it's suffixed with _rewrite.

Configurations --

Nginx --

user nginx nginx;
worker_processes 4;
events {
multi_accept on;
worker_connections 11000;
}


error_log /var/log/nginx16/nginx.log;


http {
server {
access_log off;
listen [::]:80 backlog=2 so_keepalive=60:60:0;
root /home/nginx;
server_name RHEL6;
sendfile on;
reset_timedout_connection on;
server_tokens off;
open_file_cache max=20 inactive=99999;
open_file_cache_min_uses 1;
open_file_cache_valid 99999;
open_file_cache_errors on;
log_not_found off;


rewrite ^/file1$ /0 last;
rewrite ^/file2$ /1 last;
rewrite ^/file3$ /2 last;
rewrite ^/file4$ /3 last;
rewrite "^/list([234]){0,1}/(.*)$" /$2 last;


location / {
deny all;
}
location ~ ^/[0-9]$ {
allow all;
}
location = /hello.php {
allow all;
}
location ~ ^/file[1234]$ {
allow all;
}
location ~ "^/list([234]){0,1}/[0-9]$" {
allow all;
}
}
}

Apache tuned --

ServerRoot /opt/rh/httpd24/root/usr/lib64/httpd/
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule mpm_event_module modules/mod_mpm_event.so
#LoadModule php5_module /usr/lib64/httpd/modules/libphp5-zts.so


Listen [::]:80 http
ListenBackLog 2
MaxConnectionsPerChild 0
MaxMemFree 0
ServerLimit 4
StartServers 4
# optimized for concurrency
#MaxRequestWorkers 10000
#ThreadLimit 2500
#ThreadsPerChild 2500
#MaxSpareThreads 10000
#MinSpareThreads 10000


# optimized for benchmark/HTTP header throughoutput
MaxRequestWorkers 100
ThreadLimit 25
ThreadsPerChild 25
MaxSpareThreads 100
MinSpareThreads 100


DocumentRoot /home/apache
ServerName RHEL6
User apache
Group apache
ErrorLog /var/log/httpd24/apache.log


LogLevel alert
AcceptPathInfo off
ContentDigest off
FileETag Inode Mtime
KeepAlive on
KeepAliveTimeout 60
MaxKeepAliveRequests 0
ServerTokens Full
TimeOut 5
EnableMMAP on
EnableSendfile on
ExtendedStatus off
LimitInternalRecursion 1
MaxRangeOverlaps none
MaxRangeReversals none
MergeTrailers off
Mutex pthread rewrite-map


RewriteEngine on
RewriteRule ^/file1 /0 [END,PT]
RewriteRule ^/file2 /1 [END,PT]
RewriteRule ^/file3 /2 [END,PT]
RewriteRule ^/file4 /3 [END,PT]
RewriteRule ^/list([234]){0,1}/(.*) /$2 [END,PT]


AllowOverride none
Options -FollowSymLinks
Require all denied
#SetHandler php5-script
Require all granted
Require all granted

Test commands --

echo -n list/1 list3/8 file1 list2/5 list2/0 file2 file4 list/7 list4/6 list3/3 | xargs -r -P 0 -n 1 -d ' ' -I {} /bin/bash -c 'ab -c -k -g /home/de//_$$ -s 1 -t 60 -n 9999999 -r http://[fc00::1:2]/{} &> /home/de//_stdout_$$'


echo -n {9..0} | xargs -r -P 0 -n 1 -d ' ' -I {} /bin/bash -c 'ab -c -k -g /home/de//_$$ -s 1 -t 60 -n 9999999 -r http://[fc00::1:2]/{} &> /home/de//_stdoout_$$'
The output of ab along with the report of each link served have been uploaded.

Test strategy --

Concurrency is the main thing we need to test.
We need to simulate situation when there are multiple low-bandwidth clients (all of them needs to be served concurrently to prevent some connections from being stalled). So we'll just increase the concurrent requests sent to the server using all the test machines's bandwidth.
Multiple TCP connection must be established in parallel; each of these connections will be reused to send multiple requests (keep alive will be turned on). Since the TCP connection has been established, we're not benchmarking the kernel. Speaking of which I could not get Nginx's keepalive to be turned off; that maybe the reason why Nginx was so fast in other benchmarks.
Since all webservers use sendfile() for delivering files, it's pointless to make the file large; we're not benchmarking the kernel. We're interested in how quickly the server creates HTTP headers.

Concurrency vs throughoutput (total requests/second).

Serving request serially, as opposed to concurrently is more efficient because of context switching and management overhead; we cant do anything about context switching, but the management overhead and the efficiency in constructing HTTP headers is what we want to benchmark.
But concurrency matters more than throughoutput.
If the server is independent, i.e. only it's CPU resources are used (network, disk I/o, a separate server like database are not the bottleneck like with these benchmarks), then increasing the webserver's concurrency will reduce the efficiency of the CPU cycles because of the context switching overhead. In these cases it's better to start serving another request when it finishes serving one request.
When there is a in-server server bottleneck like the network or the disk (for e.g. we'll take this e.g. for this para) and the requests are such that they take up quiet a lot of time reading the disk/network, it'll happen that the other requests timeout, or take too much time to respond. The longer the queue, more likely this'll happen. In these situations, increasing the concurrency will let the server serve multiple request in parallel sending progress to each user, abet slowly as compared to serving a single user at a time but without timing them out.
Another kind of bottleneck is towards the end user. A classic e.g. is downloading files where the client's network or the Internet is the bottle neck. If we do 1 download at a time, we wont be able to use all our hardware (disk, network etc..) to the fullest since the user's Internet connection is the bottleneck; to use it to the fullest we have to serve multiple clients. When it comes to these kind of situations, resource utilization IS about concurrency and concurrent efficiency becomes more important as the difference between the server's network speed and the client's network speed increases because that'll mean the server can serve more clients in parallel.
A similar bottleneck is when there is are multiple backend server (physical) which can handle, like, N queries in parallel. If there are X server, then the webserver must serve N*X requests in parallel to get the maximum utilization of the backend servers.

Cheating web servers --

Suppose we have a timeout of x seconds.
If the webserver is not serving the requests concurrently (to reduce context switching and management overhead and increase throughoutput), some of the requests will be within x seconds, while others will timeout.
A webserver which serves requests concurrently, will have all the requests timed out if the load exceeds a certain value.
A webserver which does not have a better concurrency is designed for benchmarks and will only perform good at benchmarks.

Apache vs Nginx in concurrency –

Nginx also appears to be serving the requests concurrently but with not as much concurrency as with Apache, but with Apache the responses were like within 21ms or lower, where as with nginx they were under 400ms; for Apache the distribution of the no. of requests served vs the interval under which they were served were not noted down for under 100ms, thus it may be cheating also.
Because of client machine limitation (it's a 10 years old machine), Apache maybe a lot faster than Nginx.

Verdict –

Apache is the clear winner.
If you switched to Nginx for the speed, your assumptions where false. Apache has a bigger toolkit, is faster and at the same time is security oriented. And in case you're wondering about the bigger CVE for Apache, it's because it has a bigger tookit and is older.
So it's all about for what purpose you tweak Apache.
Personally I don't understand the purpose of the Nginx project. If they want to optimized, they rather contribute to Apache or fork the project or create modules (something which Nginx doesn't even support) instead of creating a rival.