Main Page

From wiki.mikejung.biz
Jump to: navigation, search

Liquid web dedicated servers.gif

Wiki Landing Page

Hello there! My name is Mike Jung and this is my wiki. I hope some of the information here will help someone out or teach them something new.

Wiki Navigation and Page List

Hearthstone

Windows 8.1 Utilities and Windows Tech Preview 10

  • MPC-HC Audio Configuration - While this page focuses on configuring MPC-HC for the best audio quality, it can still be used to configure Windows 8.1 audio for high end sound cards and speakers such as the Logitech Z 906 speakers and the Sound Blaster Z sound card.
  • Windows 10 Tech Preview - Currently testing out Windows 10 Tech Preview, Build 9926. This wiki page will eventually have tons of information and updates on Windows 10.
  • CrystalDiskMark - A beginner guide on how to use CrystalDiskMark to benchmark your HDD or SSD.
  • S3 Browser - S3 Browser is similar to an FTP client, but it speaks to a REST endpoint of an S3 compatible Object Storage service. AWS S3 is supported, but other compatible Object Storage services are also supported.
  • MSI Afterburner - MSI Afterburner configuration guide / walk through with screenshots that explain how to do stuff with afterburner, like select GPU usage in OSD.
  • Virtualbox - Various tips and tricks relating to VirtualBox, which allows you to run virtual machines on Windows hosts.

GPU and Video Decoding Stuff (madVR, CUDA, MPC-HC)

  • Nvidia Noise Reduction - Nvidia Noise Reduction reduces noise during video playback, how much does this improve quality? Not a ton, in some cases, anyway, check out the page for more info.
  • Nvidia Edge Enhancement - Ever wonder if the Nvidia Video settings make much of a difference in terms of video quality? Turns out it can make a huge difference! This page focuses on Nvidia Edge Enhancements
  • MPC-HC Video Decoder Comparison - Screenshot comparisons between MPC-HC's default Video Decoder (DXVA) and CUVID. If you have a Nvidia GPU and want to make sure you have the best video quality, check out this page!
  • DyingLight - Some PC screenshots of DyingLight, using Nvidia DSR with a GTX 970. DSR does improve quality, but is very costly in terms of performance.
  • Windows 8.1 MPC-HC and MadVR Setup Guide - A Windows 8.1 based guide on how to properly configure MPC-HC (media player classic home cinema) to work with MadVR as the video renderer. This setup guide includes images for each of the steps and explanations of the main mpc-hc and madvr settings.
  • CUDA - GPUs are on track to control the global population. CUDA is how the matrix started. It's pretty cool though, CUDA allows a GPU to accelerate some types of processing that previously only the CPU could compute.
  • DXVA2 - A quick article that explains what DXVA2 is and how it interacts with a GPU during the video playback process.
  • MadVR - MadVR MPC-HC wiki containing optimization tips and benchmarks for Chroma Upscaling, Image Doubling, Image Upscaling, Image downscaling and many other configuration settings.
  • PotPlayer - PotPlayer wiki about how to install, configure and optimize potplayer using MadVR, CUDA and GPU magic
  • DirectShow - What is directshow? What does directshow do? Want to learn more about directshow? Then please visit this page!
  • MadVR Chroma Upscaling - MadVR Chroma Upscaling performance results and general information on the best scaling algorithm to use to upscale Chroma with MadVR.
  • Nnedi3 - madVR recently added the ability to use NNEDI3 to improve video quality, this wiki covers how to configure this.

cPanel Stuff

  • CloudLinux -- Overview on what CloudLinux is and the types of resources that it limits for cpanel users.
  • PHP Handlers - How to change cPanel's main PHP handler via CLI and via WHM.
  • LiquidWeb WHMCS Plugin - A FREE WHMCS plugin created by ModulesGarden that allows anyone to resell SSD VPS servers, or whatever kind of servers you want. Basically this plugin is a Cloud Business in a box and it's free.
  • CPanel Statistics Software - Learn how to configure cPanel to process logs and bandwidth during low traffic hours of the day. This reduces the amount of Apache restarts.
  • Whmcs 6 - Need help installing and configuring WHMCS 6? Visit this page!

Webserver Stuff

  • Litespeed - Information about the litespeed webserver installation process and how to correctly configure litespeed on a cpanel server.
  • Apache - Do you like websites? You can thank Apache! It's the most common webserver around. Nginx is gaining some steam, but Apache is still pretty awesome!
  • GooglePageSpeed - How to install and configure the mod_pagespeed module with Apache, in addition, and really importantly, How to install Google's mod_pagespeed on a cPanel server via Easy Apache. lol
  • WordPress Optimization - Guide on how to optimize a server for wordpress. Includes Apache Event configuration, PHP-FPM configuration, MySQL configuration and lots more!

MySQL, PHP and Caching

  • PHP 7 - This wiki page is still being worked on, but it will focus specifically on PHP 7 and how awesome it is.
  • PHP MySQL Extensions - Also still in the works, will focus specifically on the MySQL extensions for PHP.
  • Memcached - Caching makes everything faster! I like fast things, so I use memcached a lot and you should use memcached too! I'll show you how to use memcached to improve website load time and reduce latency when connecting to a database! All of this can be done if you know how to tame the mythical beast know as memcached.
  • PHP_OPcache - Do you like fast things? Want to make PHP faster? Use opcode caching. For your health!
  • fcgid - FastCGI will make ur blog faster! Maybe, if you know how to configure Apache to use FastCGI to proxy PHP requests to a dedicated PHP process! If you want to learn more about using the FCGI handler on cpanel, please check out this page!
  • Php-fpm - Speaking of awesome...PHP-FPM is here. Are you still using mod_php and wondering why apche is slow? It's because you are doing php wrong! Check out this page for information on how to install, tune, and optimize php-fpm with apache
  • Varnish - Varnish can be an excellent way to improve your website's response times by caching the most used static content, like images, varnish is able to significantly reduce the amount of work that apache has to do.
  • MySQL - Learn all about the various my.cnf settings for MySQL 5.5 and MySQL 5.6. This page is less focused on optimizations, and more focused on covering the best practices for configuring a server to handle MySQL.
  • MediaWiki Performance Tweaks - I use MediaWiki, if you can't tell already. I've created a page that lists all the caching settings / configurations to make this site somewhat fast.
  • HHVM - Facebook's attempt to take PHP to the next level. HHVM uses a JIT compiler to significantly reduce the amount of resources it takes to interpret and run a PHP script.

Monitoring and Analysis

  • Newrelic - Newrelic is pretty awesome. They offer a free tier which lets you monitor server resources for 24 hours. You can also utilize APM which is an application monitoring services which shows the response time if your application and database. If you are looking for common Newrelic agent commands or need help troubleshooting Newrelic's agents, check out this page!
  • Sysdig - Looking for a utility that will provide insight into application and Linux performance? Sysdig is your tool! I really like it and find it pretty useful so I made a wiki!
  • Sysstat - Sysstat contains sar which is used to record server resource usage over the course of each day. Sar is really helpful if you care about server performance so knowing how to view data like swap in and swap out activity is critical.

Browser and Front End

  • Chrome - A list of tweaks (flags) that you can enable in the Chrome and Chromium web browsers which can help to speed up performance. Useful if you notice slow, laggy websites and want to speed up your browser.
  • HTTP 2.0 - Still in creation mode, this wiki will eventually contain all kinds of information on the new HTTP 2.0 protocol.
  • Browser Caching - .htaccess examples for how to enable browser caching of images and static files.

Benchmarking and Performance Tuning Stuff

  • Benchmarking -- A linux benchmarking reference wiki with many example commands and explanations for sysbench, fio, iozone and ioping tests.
  • Sysbench -- Similar to the benchmarking wiki but with 100% focus on sysbench and how to benchmark vps and cloud servers.
  • Phoronix Test Suite -- How to install Phoronix Test Suite on CentOS and Ubuntu. Examples on how to use the test suite and an overview on what each test does.
  • OS Tuning - You can't tune an application until you tune the operating system. Check out my OS system tuning wiki for tips and tricks on speeding up your slow CentOS server.
  • Dmcache - Caching, SSDs, what could be better? What about using SSDs to cache your slow as balls hdds? Learn how to by checking out this dmcache wiki!
  • Siege - Learn how to compile siege so you can test website performance on http and https websites

Storage and File System Stuff

  • LVM Commands - LVM command reference guide. Explains what logical volumes and logical groups are all about and how to create an LVM volume
  • Ceph - Ceph is a distributed storage system that powers the open cloud and internet of things. Just kidding, it doesn't do all that but it is still pretty awesome technology!
  • LSI - LSI makes RAID cards. Been around for a long time, recently bought out by seagate. LSI cards are nice, but sometimes slow if you do not configure RAID for performance. If you want to add some performance to your RAID, make sure you configured the card correctly!
  • DRBD - Data replicating block device, aka DRBD has been a heavyweight in the cloud storage wars for a while now. You got Ceph in one corner, OCFS2 in another, RAID (for backups) and DRBD. DRBD can be tricky to configure and even if you get it to work it might still be somewhat slow. I have created a wiki that covers some basic performance tuning for DRBD.
  • Big Data - Main page that links to topics like Cassandra and Hadoop.

Linux Kernel Stuff

  • Software RAID - How to use mdadm and configure software RAID on Linux
  • Ubuntu - General wiki for how to do things on Ubuntu.

Other Stuff

  • Firewalld - Are you used to iptables and hate firewalld? It's ok, I feel your pain! Firewalld is actually pretty easy to use so if you're running CentOS 7 or Redhat 7 please visit this page.
  • Hearthstone Combo Guide - I'm a Hearthstone nub, so I made a wiki on how to build a deck that will at least give you some options when it comes to minion combinations.
  • Real photo postcards - Totally random old picture post card wiki. Completely unrelated to anything that has to do with the internet.
  • CentOS 7 - Main page for CentOS 7 which includes all kinds of performance tweaks for the 3.10 kernel or newer.
  • Load Balancing - Learn more about the Stingray / Riverbed Traffic Manager! It's pretty cool and has a ton of options, if you are looking for some load balancing information, check out the wiki!
  • How to save bash directory location - Bashmarks Is a simple tool that allows you to save directory locations and then later return to them using extremely simple commands that even tab complete!
  • Cassandra - Cassandra is a NOSQL like DB that Apache made. This wiki contains general information about what Cassandra is, how it works and details on the topology.
  • Hadoop - Also NoSQL like, hadoop is great for running batch jobs against a large amount of data.
  • Gcc CentOS - Why is GCC always old on CentOS? Why does CentOS always ship old software? I do not know, but I can show you how to update GCC on CentOS if you visit the GCC CENTOS wiki!
  • Hardware - Wiki that covers SSDs, PCIe Flash SSDs, RAID cards and other types of server hardware. Includes tips and tweaks to improve performance and stability.
  • Security - Basic ways to secure your server and website. How to track down malicious files and how to clean up a hacked website.
  • R1soft - Idera / R1soft backup manager overview
  • Processor - Basic information about CPUs and how they work
  • ApacheTheory - Information about how Apache works, how to optimize Apache and an explanation on the role of Apache.
  • Logs - A list of where the common cPanel log locations are and what type of information they hold.
  • Sysstat - sysstat includes sar, which is awesome

for loop example script

  • These are just basic examples of what you can do with for loops.

Locate files and do things with them

find /location/of/files -n 'file' > somelist
for each in `cat somelist` ; do something $each ; done

Sed

Add a word to the begining of a line

sed 's/^/$Wordtoadd/' original.txt > sorted_original.txt

Linux commands to check for DDoS and excessive connections

This section is being merged into the main Apache wiki which can be found here

Web Permissions | Files

Default Web Permissions NOTE: MAKE SURE YOU ARE IN A public_html directory!!!!!!

find . -type f -exec chmod 644 {} \;
find . -type d -exec chmod 755 {} \;


Find all users php.ini files.

find /home/*/public_html/* -name php.ini

Modules / Directives

speling

mod_speling.c

Once added via Easy Apache, you can simply add these directives to a .htaccess file

CheckCaseOnly On
CheckSpelling On

How To Optimize WordPress

For a detailed guide, please visit my [WordPress Optimization Guide]

PHP

Install ssh2 Pecl extension

yum install libssh2 libssh2-devel
pecl install ssh2
# You may need to update the channel, if so;
pecl channel-update pecl.php.net

Then just;
vim /etc/php.ini
extension=ssh2.so

Upload issues

May need to check two files, the first is the global php.ini file, the next is the modsec file (if applicable)

vim /usr/local/lib/php.ini

upload_tmp_dir = /tmp
session.save_path = /tmp

----------------------------------------------

vim /usr/local/apache/conf/modsec2/custom.conf

SecUploadDir /tmp
SecTmpDir /tmp

Parse Error

Parse error: syntax error, unexpected T_STRING

Check the file and remove <?xml version="1.0" encoding="utf-8"?>


Force PHP5

Add to .htaccess:

AddType application/x-httpd-php5 .html .htm

Email

How to enable DKIM for a cpanel account

  • DomainKeys Identified Mail (DKIM) defines a mechanism by which email messages can be cryptographically signed, permitting a signing domain to claim responsibility for the introduction of a message into the mail stream. Message recipients can verify the signature by querying the signer's domain directly to retrieve the appropriate public key, and thereby confirm that the message was attested to by a party in possession of the private key for the signing domain.
  • To verify that everything is setup correctly you can send an email from an email account on that domain to [email protected] No need to have a subject or body. This service will then reply with a message stating the verification of DKIM, DomainKeys, SPF, SpamAssassin, and Sender-ID. Great tool to test all kinds of email verification systems.

To install on a cPanel server:

/usr/local/cpanel/bin/dkim_keys_install <username>

or

for i in `ls /var/cpanel/users`; do /usr/local/cpanel/bin/dkim_keys_install $i; done
  • Add the Policy Record
_domainkey     IN     TXT     "t=y; o=~; n=Interim Sending Domain Policy; [email protected]"

General webmail and email permission guidelines for cPanel servers

Below are some baseline permissions that should be used with Exim and Dovecot:

/home/user/etc/

domain.com file should have:
permissions:
750
ownership:
username:mail

/home/user/etc/domain.com/

passwd
permissions:
640
ownership:
user:mail

quota
permissions:
640
ownership:
user:mail

shadow
permissions:
640
ownership:
user:user

/home/user/mail/

700 user:user cur/
751 user:user domain.com/
700 user:user anything else

If email accounts are not showing up in cPanel for a specific cpanel user be sure to check /home/$user/etc to make sure the passwd file and shadow file have proper permissions also make sure they are located in

/home/user/etc/domain.com/

If all the permissions are correct and the directories are owned by the user, try restarting cpanel mail services to see if this helps resolve the issue.


If you run into a Roundcube error like "unable to connect to database", the best thing to do is to drop the database, then re-install roundcube, which automatically re-creates the db. Make sure you backup the database before you drop it, or else you risk lots of possible data loss

cd /home/temp 
mysqldump roundcube > roundcube.sql 
mysql -e "drop database roundcube;" 
/usr/local/cpanel/bin/update-roundcube --force


If you are running into spam issues you can run the command below to find top sending IPs in exim logs:

grep "SMTP connection from" /var/log/exim_mainlog |grep "connection count" |awk '{print $7}' |cut -d ":" -f 1 |cut -d "[" -f 2 |cut -d "]" -f 1 |sort -n |uniq -c | sort -n

Find authenticated users who may be spamming:

find /var/spool/exim/input/ -name '*-H' | xargs grep 'auth_id'

Spam comming from scripts:

grep cwd=\/home\/ /var/log/exim_mainlog| cut -d' ' -f4 | sort | uniq -c | sort -n

Removing all queued messages at once in a safe way:

exim -bp | awk '/^ *[0-9]+[mhd]/{print "exim -Mrm " $3}' | sh

Or you can do the same from the mail queue manager in WHM.


APF SMTP tweak enables mail to be sent only from the mail or mailman GID, and blocks all outbound SMTP, except through the sendmail binary. Add this bold line of code to /etc/init.d/apf , right underneath the start) case:

/usr/local/sbin/apf --start >> /dev/null 2>&1
'''/scripts/smtpmailgidonly on'''
echo_success

FTP

If you are having issues with Proftp connections or with authentication. Check the Proftp configuration file below and make sure that "AuthPAM" is actually on.

vim /etc/proftpd.conf
AuthPAM on

If you want to make sure PureFTP is using FTPES, edit /etc/pure-ftpd.conf and uncomment (enable) the PassivePortRange line, like below.

# Port range for passive connections replies. - for firewalling.
PassivePortRange          30000 50000

APF - /etc/apf/conf.apf

# Common ingress (inbound) TCP ports
IG_TCP_CPORTS="20,21,22,25,53,80,110,143,443,465,993,995,2082,2083,2084,2086,2087,2095,2096,3306,6666,7786,30000_50000"

# Common egress (outbound) TCP ports
EG_TCP_CPORTS="21,25,80,443,43,30000_50000"


CSF - /etc/csf/csf.conf

# Allow incoming TCP ports
TCP_IN = "20,21,22,25,53,80,110,143,443,465,953,993,995,2077,2078,2082,2083,2086,2087,2095,2096,30000:50000"

# Allow outgoing TCP ports
TCP_OUT = "20,21,22,25,37,43,53,80,110,113,443,587,873,953,2087,2089,2703,30000:50000"


If you are encountering vsftp timeout issues or strange dns like issues with vsftp check the vsftpd configuration file and make sure that reverse_lookup_enable is set to no

/etc/vsftpd/vsftpd.conf:

reverse_lookup_enable=NO

Nginx

Common configuration settings

  • The main configuration file to edit is /etc/nginx/nginx.conf, which by default also reaches out to include any additional configuration files in the conf.d directory and any virtual host files in the sites-enabled directory.
  • worker_processes in /etc/nginx/nginx.conf. This should be equal to the amount of CPU cores the server has.
worker_processes $CPUs;
  • worker_connections defines how many connections each worker process is allowed to handle
  • worker_processes x worker_connections tells the maximum amount of HTTP connections possible at any moment

File cache settings

http {
[...]
        ##
        # File Cache Settings
        ##

        open_file_cache          max=5000  inactive=20s;
        open_file_cache_valid    30s;
        open_file_cache_min_uses 2;
        open_file_cache_errors   on;

Gzip This will compress content at the expense of a little extra CPU, but it will save a lot of bandwidth.

gzip on;
gzip_disable "msie6";
gzip_min_length 1100;
gzip_vary on;
gzip_proxied any;
gzip_buffers 16 8k;
gzip_types text/plain text/css application/json application/x-javascript
    text/xml application/xml application/rss+xml text/javascript
    image/svg+xml application/x-font-ttf font/opentype
    application/vnd.ms-fontobject;

Conflicting Server Name Error

Check for duplicates/system users:

grep -i domain.com /var/cpanel/users/*

If there is a domain entry owned by "system" remove this file:

rm /var/cpanel/users/system

Then run:

/scripts/rebuildnginxvhost

cPanel Tips and Tricks

httpd.conf domain errors?

info [rebuildhttpdconf] Unable to determine group for $username, skipping domain $domain.com

Check /var/cpanel/userdata/$user/$domain.com
Make sure group: is set correctly
/scripts/rebuildhttpdconf
service httpd restart

Exclude files from being updated.

vim /etc/cpanelsync.exclude

Then add the absolute path for the file. An example would be Roundcube webmail settings:

/usr/local/cpanel/base/3rdparty/roundcube/config/main.inc.php

Databases listed in Cpanel, but do not actually exist

Check the following files and remove any users / dbs that do not exist:

/var/cpanel/databases/

$user.cache
$user.yaml    

spamd issues

/scripts/perlinstaller IO::Socket::IP --force

DNS

Disable zone transfers with named.conf

acl can_axfr {
127.0.0.1;
};

options {
    allow-recursion { trusted; };
    allow-transfer { can_axfr; };
};

WARNING: key file (/etc/rndc.key)

service named stop
mv /etc/rndc.conf /etc/rndc.conf.OLD
service named start

NFS

yum install nfs*
mkdir /$whatever/you/want/to/share

vim /etc/exports
added:
/$whatever/you/want/to/share       $IPADDY/Subnetmask(rw,no_root_squash,subtree_check)

/etc/init.d/nfs start
/etc/init.d/nfslock start
/etc/init.d/rpcbind start
/etc/init.d/rpcidmapd restart

vim /etc/idmapd.conf
Uncommented / added:
Domain = $local.domain.com

chkconfig rpcbind on
chkconfig rpcidmapd on
chkconfig nfs on
chkconfig nfslock on

Make sure port 2049 is open as well. 

IPTABLES

This is an example of a default IPTABLES set of rules:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
:TRUSTED - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
###Add trusted IPs / hosts / IP blocks here
###Example would be:
-A TRUSTED -s 192.168.0.0/24
-A TRUSTED -s $myhomeIP
-A TRUSTED -s $someotherserver
###END TRUSTED HOSTS SECTION
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 53 -j ACCEPT
###EXAMPLE FOR ACTIVE/PASSIVE FTP ACCESS FOR TRUSTED HOSTS
-A RH-Firewall-1-INPUT -p tcp --dport 21 -j TRUSTED
-A RH-Firewall-1-INPUT -p tcp --dport 20 -j TRUSTED
-A RH-Firewall-1-INPUT -p tcp --dport 30000:50000 -j TRUSTED
###END FTP EXAMPLE
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited


Kernel Stuff

Tools and Utilities used to build a kernel

gcc --version
  • Used to compile the kernel
ld -v
  • Tools used to assist when compiling the kernel
make --version
  • Used to determine which files are needed to compile the kernel

Tools and Utilities to use the kernel

fdformat --version
  • Used to handle mounting of disks
depmod -V
  • Used to load kernel modules and remove them

File System Tools

tune2fs
  • Used to handle the file systems such as ext4


Command to see what modules are loaded:

lsmod

See all modules, even if they are not loaded:

modprobe -l 

Get detailed information on a module:

modinfo $module

Remove a module (assuming no other dependents are using it):

modprobe -r $module 

See all kernel settings

sysctl -a

TCP_FIN_TIMEOUT This setting determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. During this TIME_WAIT state, reopening the connection to the client costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, making more resources available for new connections. Addjust this in the presense of many connections sitting in the TIME_WAIT state:

# echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
(default: 60 seconds, recommended 15-30 seconds)

Steps to compile and customize a kernel

The steps below will download the kernel source, decompress it, then will make the kernel with the default options.

mkdir $place to put the kernel
cd $place to put the kernel
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.9.tar.xz
xz -d linux-3.9.tar.xz
tar -xvf linux-3.9.tar
cd linux-3.9/
make defconfig

From here, we can customize the kernel further.

make menuconfig

Options when using menuconfig:

[*] = Selected, if no star then not selected
<Y> = Select module to be built into the Kernel
<M> = Select module to be built as a module to be loaded, but not built into the kernel

Postfix

Log location:

/usr/local/psa/var/log/maillog


Some one liners to figure out what is in the queue and how to remove bullshit emails.

mailq | grep ^[A-Z\|0-9] | awk '{print $7}' | cut -d@ -f2 | sort | uniq -c | sort -rn | head -15

Once you figure out senders or whatever, you can do something like this to either delete the email or put it in the hold queue

Put in hold queue

mailq | grep $someshittydomain.com | awk '{print $1}' |  postsuper -h -

Delete the emails

mailq | grep $someshittydomain.com | awk '{print $1}' |  postsuper -d -

If these commands dont remove all the emails, you might need to use cut to get rid of the "!" or "*" which sometimes get placed at the end of the email id

Benchmarking Tools

Please visit this page for more up to date information

ZFS

This section is based off of an excellent guide by Ars.


Creating ZFS Pool

This will list available devices to use

ls -l /dev/disk/by-id

Once you determine what devices to use, this command will create the pool

zpool create -o ashift=12 $name $raidz_type /dev/disk/by-id/$$ /dev/disk/by-id/$$ /dev/disk/by-id/$$

NOTE

  • -o ashift=12 means "use 4K blocksizes instead of the default 512 byte blocksizes," which is appropriate on almost all modern drives.

ZFS Commands

This will display raw capacity status

zpool list

This will display usable status

zfs list

You can create "filesystems" which are much like pre-formated paritions or folders.

zfs create $zfs_vol/$folder_name

You can and should create multiple filesystems so that you can manage each partition individually. If you have groups of content that you seperate already, then it makes sense to create multiple filesystems, such as images, movies, txt files, etc. By doing this you can take advantage of ZFS's settings.

zfs set compression=on $zfs_vol/textfiles
zfs set quota=200G $zfs_vol/jpegs

View CPU Temps in Cent 6.5

For most new CPUs and Mobos this should be pretty simple to do. For this example, I'm using a newer SuperMicro Motherboard.

## Install the package

yum -y install lm_sensors

## Detect the sensors, should be fine to say YES to all the questions

sensors-detect

## If everything installed correctly, you should see all the CPU core temps

sensors


Example output, for this example I am using an Intel E5-1650v2

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0: +47.0°C  (high = +80.0°C, crit = +90.0°C)  
Core 0:        +47.0°C  (high = +80.0°C, crit = +90.0°C)  
Core 1:        +44.0°C  (high = +80.0°C, crit = +90.0°C)  
Core 2:        +41.0°C  (high = +80.0°C, crit = +90.0°C)  
Core 3:        +40.0°C  (high = +80.0°C, crit = +90.0°C)  
Core 4:        +40.0°C  (high = +80.0°C, crit = +90.0°C)  
Core 5:        +39.0°C  (high = +80.0°C, crit = +90.0°C)  

Linux Memory Usage Overview

There are two commonly displayed values for Linux RAM usage. When using a tool like ps, you often times see VSZ and RSS.

VSZ: "VSZ is the Virtual Memory Size. It includes all memory that the process can access, including memory that is swapped out and memory that is from shared libraries. "

RSS: "RSS is the Resident Set Size and is used to show how much memory is allocated to that process and is in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It does include all stack and heap memory.

  • RSS And VSZ do not accurately represent the real RAM usage for a process, they report the total RAM the process would use if it were the only process running, but many processes share memory if they use the same shared libraries.
  • Shared libraries like libc are commonly used by many different applications, Linux is able to load the library once into RAM, and then multiple processes can re-use the same library at the same time without having to duplicate the library which would use more RAM. Linux is very efficient because of its ability to share libraries among many processes.


You can use pmap to get more specific memory usage information from a process.

pmap -d $PID

An example command is:

pmap -d 15441
Address           Kbytes Mode  Offset           Device    Mapping
....
....
00007f574e0a4000       8 rw--- 0000000000003000 0fc:00003 cStringIO.so
00007f574e0a6000      20 r-x-- 0000000000000000 0fc:00003 stropmodule.so
00007f574e0ab000    2044 ----- 0000000000005000 0fc:00003 stropmodule.so
00007f574e2aa000       8 rw--- 0000000000004000 0fc:00003 stropmodule.so
00007f574e2ac000      12 r-x-- 0000000000000000 0fc:00003 timemodule.so
00007f574e2af000    2048 ----- 0000000000003000 0fc:00003 timemodule.so
00007f574e4af000       8 rw--- 0000000000003000 0fc:00003 timemodule.so
00007f5754477000     540 rw--- 0000000000000000 000:00000   [ anon ]
00007f5754507000      12 rw--- 0000000000000000 000:00000   [ anon ]
00007fff09ca1000     112 rw--- 0000000000000000 000:00000   [ stack ]
00007fff09dff000       4 r-x-- 0000000000000000 000:00000   [ anon ]
ffffffffff600000       4 r-x-- 0000000000000000 000:00000   [ anon ]
mapped: 196340K    writeable/private: 9372K    shared: 0K
  • The lines that have "r-x--" are considered the code segments.
  • The lines that have "rw---" are considered the data segments.
  • The important information here is the "writeable/private" value, which is the incremental cost of the process once you remove all the other shared libraries that were already loaded / can be used by other processes.

Using an Apache process for another example:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
nobody   22696  0.0  4.9 649624 49548 ?        Sl   17:45   0:01  \_ /usr/local/apache/bin/httpd -k start -DSSL
  • VSZ reports 649624K, or about 634MB
  • RSS reports 49548K, or about 48MB

Running pmap on that PID we see:

pmap -d 22696
....
....
mapped: 649624K    writeable/private: 63292K    shared: 184140K
  • writeable/private: 63292K, or around 63MB, you can see that much of this process is using shared libraries.

Storm and LiquidWeb API

You can find API documentation at the link listed below.

If you have issues using the Liquid Web API the first step would be to run a simple curl command to make sure you can connect to the API and that are you using the correct user name and password. Please replace $API_USER and $API_PASS with your credentials. PLEASE be aware that this is not the most secure way to test this, you might want to throw this command into a file and run it that way, otherwise your credentials will be on the server's history, obviously this is not preferred.' You can create a temporary API user just to test, then remove the user or update the password.

curl https://$API_USER:[email protected]/v1/utilities/info/ping.json

Docker run command line examples

This command will run a container in interactive mode and will put you in the container as soon as it is started.

docker run -i -t -p $IP:$HostPort:$ContainerPort -v $HostDirectory:$ContainerDirectory $Image $Command

An Example Command would be if you wanted to run a container with Apache that listens on port 80 in the container, and port 9000 on the host. We will also have the container use a directory on the host so that data persists even if the container is stopped or killed

docker run -p 8.8.8.8:9000:80 -v /partition1:/parition1 doge/apache:latest /usr/sbin/apache2ctl -D FOREGROUND

Quick and Dirty script to KILL off all containers

for each in `docker ps | awk '{print $1}'` ; do docker kill $each ; done

Quick and Dirty script to STOP all containers, this is slower than the above command

for each in `docker ps | awk '{print $1}'` ; do docker stop $each ; done

Linux Kernel Networking

A really good article that explains how networking performance in the Linux kernel will need some improvements in the near future. - https://lwn.net/Articles/629155/