FreeNas, Transmission and Corrupted Files – Part 1

After many years of just ‘taking’ I mature and upgraded my setup to be able to give community back what I took.

I setup RAID 6 FreeNas machine. Thanks to playing with those I understood why everybody in FreeNas community hate Realtek and why people just say to throw it away and get something that works (FreeNas: Realtek 8111F and re0: Watchdog Timeout Error and FreeNas: Realtek 8111F and “re0: Watchdog Timeout Error” – universal solution). In the end I join the ‘hate club’ and bought Intel PRO 1000 PT Dual Port NIC. Fun part is that Linux handle Realtek with ease but I been with Linux when It didn’t like most of hardware that Windows support so I can understand that there are still hardware issues with FreeBSD.

Cool, FreeNAS install without any issues.

If you unfamiliar with FreeNAS it comes with ‘plugins’ that are basically jails with pre-configured application that ‘just works’.

I slap that Install button and almost instantaneous I had working Transmission 2.93 (FreeNAS should work about pushing updates to plugins faster but yeah, nothing to complicate that we cannot fix ourself if needed)

I mounted needed directories, slap few torrent files that I already had there and enjoyed how transmission is handling all those sweet, sweet files out there.

And Tranmission was working and it was doing good. But after many, many torrent more that join the seeding happy farm not many people did connect to share the love from me. I started tweaking Transmission. Setting by setting I scope documentation looking for holy grail, some magic setting that would boost simultaneous connections. I found few. So I created new config file, and it was good. Not the best but I notice some connection increase. But I was thinking that maybe its because the application is inside jail aka virtual network interface. Few threads on popular forums, many many blog post later I saw the solution allow.raw_sockets=true. Great I saw a big improvement with connections – but who know if that was really all this tweaking doing or just someone join the swarm.

Few days later I notice big (I could even say tremendous) packet drop from/in my network. What was happening ? Router that is used between NAS and WAN have working QoS rules so it shouldn’t be a problem with using to much of bandwidth… So what was it ?

Culprit here was the CPU inside the router – we need to remember that using fancy feature that make router even smarter use CPU. QoS, Firewall, VPN all of them take some of that sweet-sweet pie (CPU). But why its dropping packets you ask? Simple, there is less and less CPU power to handle all the traffic that is going to and from network so everything is super slow. But wait, didn’t it work before Transmission? Yes it did – and we could disable some functions but we can do better. Lets optimize some firewall rules, we can cut QoS from 10 types to 5 without big sacrifices. Is that enough? No, but we can tweak both Transmission and firewall rule for it. Let set ‘Max connections’ to smaller number and double that with same (or 1% more on firewall rule so we are double sure that anything that is marked as ‘new connection’ and is about our limit get dropped).

And this game enough room for that sweet-sweet pie named CPU could handle ‘few’ packets more.

And it was good for months and months…

Until the corruption start to show !

(will continue in part 2)

MySQL row size too large (>8126)

This is something to do with bad table design, where you are using multiply columns with Text values, the best solution would be redesign tables, but from time to time you need to patch it at-hoc without redesigning application that use that database.

Solution that usually works:

# to my.conf mysql configuration file add/replace:
innodb_file_per_table
innodb_file_format = Barracuda

# and alter table that throw error with
ALTER TABLE your_table ENGINE=InnoDB ROW_FORMAT=DYNAMIC

# to my.conf mysql configuration file add/replace:

innodb_file_per_table

innodb_file_format = Barracuda

# and alter table that throw error with

ALTER TABLE your_table ENGINE=InnoDB ROW_FORMAT=DYNAMIC

Export or save Putty configuration

From time to time we need migrate our putty configuration/sessions that we saved (configurations, IPs etc).

Putty store those data inside system register and those are some methods to extract them and save to new pc:

# cmd
regedit /e "%USERPROFILE%\Desktop\putty.reg" HKEY_CURRENT_USER\Software\SimonTatham

# powershell
reg export HKCU\Software\SimonTatham ([Environment]::GetFolderPath("Desktop") + "\putty.reg")

# regedit
HKEY_CURRENT_USER -> Software -> SimonTatham -> PuTTY

# cmd

regedit /e "%USERPROFILE%\Desktop\putty.reg" HKEY_CURRENT_USER\Software\SimonTatham

# powershell

reg export HKCU\Software\SimonTatham ([Environment]::GetFolderPath("Desktop") + "\putty.reg")

# regedit

HKEY_CURRENT_USER -> Software -> SimonTatham -> PuTTY

We end up with putty.reg on our Desktop which we can execute on new computer and enjoy with same settings/sessions as before.

Nagios – network/device configuration

If you don’t have working Nagios installation fell free to follow Nagios – New installation on Debian 9 link.

We will need to add folder structure (for better organization) to nagios config file:

# edit nagios.cfg
mcedit /usr/local/nagios/etc/nagios.cfg

# lets add out root dir that will store our config files for device/network
# this way we don't need to type cfg_file for each .cfg file out there
cfg_dir=/usr/local/nagios/etc/devices

# save file and create proper directory
mkdir /usr/local/nagios/etc/devices
cd /usr/local/nagios/etc/devices

# edit nagios.cfg

mcedit /usr/local/nagios/etc/nagios.cfg

# lets add out root dir that will store our config files for device/network

# this way we don't need to type cfg_file for each .cfg file out there

cfg_dir=/usr/local/nagios/etc/devices

# save file and create proper directory

mkdir /usr/local/nagios/etc/devices

cd /usr/local/nagios/etc/devices

Not everyone have luxury to always use smart switches around network – those make live a lot easier, but life is not a dream 🙂

Let us make a template for non-smart aka no-ping switches:

define host{
        name                            noping-host    ; The name of this host template
        notifications_enabled           1       
        event_handler_enabled           1       
        flap_detection_enabled          1      
        process_perf_data               1      
        retain_status_information       1       
        retain_nonstatus_information    1       
        notification_interval           0
        notification_period             24x7
        notification_options            d,u,r
        contact_groups                  admins
        max_check_attempts              5
        address                         127.0.0.1      ; Just so we don't have to write it all the time
        register                        0      
        }

define host{

name noping-host ; The name of this host template

notifications_enabled 1

event_handler_enabled 1

flap_detection_enabled 1

process_perf_data 1

retain_status_information 1

retain_nonstatus_information 1

notification_interval 0

notification_period 24x7

notification_options d,u,r

contact_groups admins

max_check_attempts 5

address 127.0.0.1 ; Just so we don't have to write it all the time

}

Now we can define few switches like this:

define host{
            use noping-host
            host_name switch-number-1
            alias Switch Number 1
            icon_image switch40.png
            statusmap_image switch40.gd2
            parent name_of_parent_host_name  ; use this to make tree on map
            initial_status u
}

define host{

use noping-host

host_name switch-number-1

alias Switch Number 1

icon_image switch40.png

statusmap_image switch40.gd2

parent name_of_parent_host_name ; use this to make tree on map

initial_status u

}

You can most variables put to template file so there will be even less writing

Most common type of templete is: linux-server (you can check others in /usr/local/nagios/etc/objects/templates.cfg )

After making few .cfg files inside our directory remamber to chown nagios:nagios -R /usr/local/nagios/etc/devices/ to avoid erros and restart nagios

First steps with new server

#change password
passwd

#update ubuntu/debian
apt-get update
apt-get upgrade

#install most needed packages in ubuntu/debian
apt-get install sudo openssl openssh-server fail2ban sudo

#add user to use for ssh connection (replace *user* with your own user name)
adduser *user*

#copy rules for fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

#edit
vim /etc/fail2ban/jail.local

#add or replace content in file to resemble:
[ssh-ddos]
enabled = true

#reset
/etc/init.d/fail2ban restart

#disable root access via ssh
mkdir --parents /home/*user*/.ssh
chown *user*:*user* /home/*user*/.ssh/
sed -i -e 's/PermitRootLogin.*/PermitRootLogin no/' '/etc/ssh/sshd_config'
/etc/init.d/ssh reload

#enable ssh connection only for selected user group
addgroup --system "ssh-users"
command echo 'AllowGroups ssh-users' >> /etc/ssh/sshd_config
/etc/init.d/ssh reload
adduser *user* ssh-users

#generate keys for ssh connection
#on client computer (machine that you will use to establish connection with server)
ssh-keygen -t rsa -f /home/*user*/.ssh/id_rsa
ssh-copy-id -i "/home/*localuser*/.ssh/id_rsa.pub" *user*@server

#on server
sed -i -e 's/^[#\t ]*PubkeyAuthentication[\t ]*.*$/PubkeyAuthentication yes/' '/etc/ssh/sshd_config'
/etc/init.d/ssh reload

#we can set custom ports for ssh server (ex. port 1234 instead 22)
command sed -i -e "s/^[#\t ]*Port[\t ]*.*\$/Port 1234/" '/etc/ssh/sshd_config'
/etc/init.d/ssh reload

#we edit rule for ssh inside fail2ban file:
echo "[ssh]
port = 1234" >> '/etc/fail2ban/jail.local'
/etc/init.d/fail2ban restart

#Profit!

#change password

passwd

#update ubuntu/debian

apt-get update

apt-get upgrade

#install most needed packages in ubuntu/debian

apt-get install sudo openssl openssh-server fail2ban sudo

#add user to use for ssh connection (replace *user* with your own user name)

adduser *user*

#copy rules for fail2ban

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

#edit

vim /etc/fail2ban/jail.local

#add or replace content in file to resemble:

[ssh-ddos]

enabled = true

#reset

/etc/init.d/fail2ban restart

#disable root access via ssh

mkdir --parents /home/*user*/.ssh

chown *user*:*user* /home/*user*/.ssh/

sed -i -e 's/PermitRootLogin.*/PermitRootLogin no/' '/etc/ssh/sshd_config'

/etc/init.d/ssh reload

#enable ssh connection only for selected user group

addgroup --system "ssh-users"

command echo 'AllowGroups ssh-users' >> /etc/ssh/sshd_config

/etc/init.d/ssh reload

adduser *user* ssh-users

#generate keys for ssh connection

#on client computer (machine that you will use to establish connection with server)

ssh-keygen -t rsa -f /home/*user*/.ssh/id_rsa

ssh-copy-id -i "/home/*localuser*/.ssh/id_rsa.pub" *user*@server

#on server

sed -i -e 's/^[#\t ]*PubkeyAuthentication[\t ]*.*$/PubkeyAuthentication yes/' '/etc/ssh/sshd_config'

/etc/init.d/ssh reload

#we can set custom ports for ssh server (ex. port 1234 instead 22)

command sed -i -e "s/^[#\t ]*Port[\t ]*.*\$/Port 1234/" '/etc/ssh/sshd_config'

/etc/init.d/ssh reload

#we edit rule for ssh inside fail2ban file:

echo "[ssh]

port = 1234" >> '/etc/fail2ban/jail.local'

/etc/init.d/fail2ban restart

#Profit!

M	T	W	T	F	S	S
« Jul
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31