CloudLinux

From wiki.mikejung.biz
Jump to: navigation, search

Liquidweb 728x90.jpg

CloudLinux Resource Limits Explanation

CloudLinux helps to control resources on a cPanel server. It does this by limiting resources that are broken up into a few different categories such as Memory, Process Limits and CPU usage. Below are all the different limits that you can configure with CloudLinux.

CloudLinux Virtual(VMEM) and Physical(PMEM) Memory Limit

CloudLinux uses virtual (VMEM) and physical (PMEM) memory limits.

  • Virtual (VMEM) can be viewed via top or any other resource reporting utility as VIRT. The virtual memory used is almost always way more than physical memory used because virtual memory does not factor in shared memory which can be used by many processes at the same time, nor does virtual memory limits factor in disk cache usage. Because of this CloudLinux suggest you use Physical Memory Limits instead of virtual memory limits for shared hosting environments.
  • Physical PMEM) limits the amount of physical memory that is actually being used by the LVE user's processes. This shows up in the RES column in TOP or when using PS. CloudLinux suggests using PMEM to limit memory usage in shared environments and I have no idea why you would use CloudLinux if you were not splitting up resources, so you should always use the PMEM limit with CloudLinux

If you want to give a specific LVE user 1GB of Physical Memory to use, you would use the command below.

lvectl set $ID --pmem=1G

If you only want to give LVE 512MB RAM

lvectl set $ID --pmem=512M

Official CloudLinux Memory Limit Documentation

CloudLinux CPU Limit

The CPU limit is considered to be deprecated after LVE-UTILS 1.4. It's been replaced by the SPEED limit which is covered below. If you do not see SPEED limit then you should upgrade CloudLinux

DEPRECATED CPU DOCS

CloudLinux SPEED Limit

The CloudLinux SPEED Limit enforces the amount of CPU each LVE can use. You can set this limit as a % of a single CPU or you can specify a fixed amount of Mhz / Ghz to use, so if you wanted to split up LVE resources based off CPU count you can, or if you want to slice up CPU based off of total mhz you could also do that.

If you want to set a specific LVE to be limited to 1 CPU core, you would use -speed=100%

lvectl set $ID --speed=100%

If you wanted to give a specific LVE access to 4 CPU cores you could use

lvectl set $ID --speed=400%

If you have multiple servers with mixed hardware you could specify a GHz value to try and make CPU resources even between servers with very different CPUs. If the CPU being used is 4 GHz and I give an LVE a 2 GHz limit that would effectively be the same as giving that user access to 0.50 CPUs. This setting seems way more complicated than it needs to be and I suggest sticking to the percentage based approach instead of using the ghz or mhz approach.

lvectl set $ID --speed=2ghz

Official CloudLinux SPEED Limit Documentation

CloudLinux IOPS Limit

You can use CloudLinux lvectl to modify other limits besides just entry processes. You can also modify per LVE disk IOPS limits. This only effects IO coming to or from the disk, the iops limit for CloudLinux does not count cached reads as IOPS so if most reads are cached in RAM (mysql data for instance) that will not be counted toward this limit. If each IO is 4KB (pretty common with most Linux File systems), you are also setting the throughput, or io limit covered in the section below. 1000 IOPS @ 4KB block size per request equals about 4000KB/s or 4MB/s

lvectl set $ID --iops=1024

Official CloudLinux IOPS Limit Documentation

If you know that your storage can handle up to say, 10,000 IOPs and you want to evenly slice up the total IO among 10 LVEs you would give each LVE a limit of about 1000 IOPS, probably more since most of the time you don't see 10 users all using 100% IO all the time.

You can view what the current IOPS limit is for a specific CloudLinux LVE by running the command below. Replace $LVE with the actual LVE ID you want to view

lvectl limits $LVE

CloudLinux IO Limit

In addition to settings IOPS limits, you can also limit IO throughput. If you allow for 1000 IOPS, and we assume the block size per IO is 4KB then you need to set this value to be higher than the IOPS limit. Basically whatever limit is lowest will limit the other one, at least that's how I understand it. To limit an LVE user to 1000 IOPs (4MB/s) you could set this value to 4000. The LVE IO limit only effects DISK IO, this value does not limit network IO.

lvectl set $ID --io=4000KB/s

Official CloudLinux IO Limit Documentation

You can view what the current IO limit is for a specific CloudLinux LVE by running the command below. Replace $LVE with the actual LVE ID you want to view

lvectl limits $LVE

CloudLinux EP (Entry Processes Limit)

To view a user's current limits you can use the command, replacing $LVE with the correct user ID. In this example you can see that EP is set to 25. The CloudLinux EP limit is basically the same as Apache Max Clients or FCGI max processes per class. If the user reaches this limit, and has 25 PHP processes running, and 1 more request comes in, it will fail / not complete until one of the currently running 25 PHP processes exits.

lvectl limits $LVE
      ID   SPEED    NCPU    PMEM    VMEM      EP   NPROC      IO    IOPS
     $ID     100      16    1.0G    2.0G      25       0    1024    1024

If EP is too low and you experience slow website loading times / errors you will want to make sure the EP limit is not set too low. For most websites values of 25 or less should be more than enough.


To raise the maxEntryProcs limit for a specific CloudLinux user, you can use the lvectl command to raise or lower this limit.

lvectl set $ID --maxEntryProcs=25

Official EP CloudLinux Doc Page

CloudLinux NPROC (Number of Processes Limit)

The CloudLinux NPROC Limit sets a hard cap for the amount of processes that can run inside of a single LVE at once. Once this limit has been reached new processes cannot be created until one of the existing processes dies. This value only counts processes and not threads so if a single HTTPD processes is created inside an LVE and it creates a child thread that still only counts for 1 NPROC.

I suggest setting EP or EntryProcessLimits instead of NPROC as EP will do a good job at limiting overall processes. If you do want to set or adjust NPROC you can do so by running this command

lvectl set $LVE_ID --nproc=1000

Again, I suggest setting nproc to 0 (unlimited) and configuring the entry process limit to be something like 20 or 30 and use the EP limit instead of nproc.

CloudLinux CLI Tools and Command Line Examples

lveinfo

lveinfo queries stats from the lve-stats package which allows you to get a much better idea on LVE usage. For instance if you wanted to query all LVE users ordered by their IO usage you would run the command below. In this case there is only one LVE, but if you had 100 LVEs or more the top IO users will be at the top. The io_max value does not help to find consistently high IO users, only the highest bursts of IO so if you want to see average IO then use -o io_avg

lveinfo -o io_max

+---+----+----+----+---+---+----+---+---+---+-----+-----+-----+-----+-----+-----+---+-----+-----+-----+-----+-----+
| ID|aCPU|mCPU|lCPU|aIO|mIO|lIO |aEP|mEP|lEP|aVMem|mVMem|lVMem|aPMem|mPMem|lPMem|EPf|VMemF|PMemF|aIOPS|mIOPS|lIOPS|
+---+----+----+----+---+---+----+---+---+---+-----+-----+-----+-----+-----+-----+---+-----+-----+-----+-----+-----+
|501| 0  | 0  | 7  | 0 | 0 |1024| 0 | 0 | 20|  0  |  0  | 2.0G| 8.7M| 8.7M| 1.0G| 0 |  0  |  0  |  0  |  0  | 1024|
+---+----+----+----+---+---+----+---+---+---+-----+-----+-----+-----+-----+-----+---+-----+-----+-----+-----+-----+

For lveinfo version 0.8 these are the valid IDs and explainations.

ID                     : LVE Id or username
aCPU                   : Average CPU usage
mCPU                   : Max CPU usage
lCPU                   : CPU Limit
aEP                    : Average Entry Processes
mEP                    : Max Entry Processes
lEP                    : maxEntryProc limit
aVMem                  : Average Virtual Memory Usage
mVMem                  : Max Virtual Memory Usage
lVMem                  : Virtual Memory Limit
VMemF                  : Out Of Memory Faults
EPf                    : Entry processes faults
aPMem                  : Average Physical Memory Usage
mPMem                  : Max Physical Memory Usage
lPMem                  : Physical Memory Limit
aNproc                 : Average Number of processes
mNproc                 : Max Number of processes
lNproc                 : Limit of Number of processes
PMemF                  : Out Of Physical Memory Faults
NprocF                 : Number of processes faults
aIO                    : Average I/O
mIO                    : Max I/O
lIO                    : I/O Limit
aIOPS                  : Average I/O Operations
mIOPS                  : Max I/O Operations
lIOPS                  : I/O Operations Limit

lvectl

To raise the maxEntryProcs limit for a specific CloudLinux user, you can use the lvectl command to raise or lower this limit.

lvectl set $ID --maxEntryProcs=25

If maxEntryProcs is set to say, 10, for a single user, and you allow them 20 PHP processes with FCGI or PHP-FPM or whatever, the user will only be allowed to have up to 10 PHP processes running at once. To remove this possible bottleneck you want to make sure the CloudLinux EP limit is equal to the amount of concurrent PHP processes you want to the user to be able to run.

If a user reaches this limit then HTTP responses will begin to fail until older processes under that LVE fail.

Official CloudLinux EP Limit Documentation

You can use CloudLinux lvectl to modify other limits besides just entry processes. You can also modify per LVE disk IOPS limits. This only effects IO coming to or from the disk, the iops limit for CloudLinux does not count cached reads as IOPS so if most reads are cached in RAM (mysql data for instance) that will not be counted toward this limit. If each IO is 4KB (pretty common with most Linux Filesystems), you are also setting the throughput, or io limit covered in the section below. 1000 IOPS @ 4KB block size per request equals about 4000KB/s or 4MB/s

lvectl set $ID --iops=1024

Official CloudLinux IOPS Limit Documentation

If you know that your storage can handle up to say, 10,000 IOPs and you want to evenly slice up the total IO among 10 LVEs you would give each LVE a limit of about 1000 IOPS, probably more since most of the time you don't see 10 users all using 100% IO all the time.

In addition to settings IOPS limits, you can also limit IO throughput. If you allow for 1000 IOPS, and we assume the block size per IO is 4KB then you need to set this value to be higher than the IOPS limit. Basically whatever limit is lowest will limit the other one, at least that's how I understand it. To limit an LVE user to 1000 IOPs (4MB/s) you could set this value to 4000. The LVE IO limit only effects DISK IO, this value does not limit network IO.

lvectl set $ID --io=4000KB/s

Here are all the possible command line options for lvectl version 1.4-32

lvectl --help
lvectl version 1.4-32.el6.cloudlinux
usage: lvectl command [veid] [options]
commands:
apply                    apply config settings to specified LVE
apply all                apply config settings to all the LVEs
apply-many               to apply LVE limits to multiple distinct LVEs (uids of users are read from stdin)
set                      set parameters for a LVE and/or create a LVE
set-user                 set parameters for a LVE and/or create a LVE using username instead of ID
list                     list loaded LVEs
list-user                list loaded LVEs, display username instead of user id
limits                   show limits for loaded LVEs
delete                   delete LVE and set configuration for that LVE to defaults
delete-user              delete LVE and set configuration for that user to defaults
destroy                  destroy LVE (configuration file remains unchanged)
destroy all              destroy all LVE (configuration file remains unchanged)
destroy-many             to destroy LVE limits to multiple distinct LVEs (uids of users are read from stdin)
package-set              set LVE parameters for a package
package-list             list LVE parameters for packages
package-delete           delete LVE parameters for a package
paneluserslimits         show current user's limits for control panel
limit                    limit PID into specified LVE. Parameters PID LVE_ID
release                  release PID from LVE. Parameters PID
set-binary               add binary to be run inside LVE on execution
del-binary               remove binary from being run inside LVE on execution
list-binaries            list all binaries to be run inside LVE on execution
load-binaries            load binaries (used on startup) from config file
reload-binaries          re-load list of binaries from config file
help (-h)                show this message
version (-v)             version number
lve-version              lve version number
options:
--cpu=N                  limit CPU usage; (deprecated. Use --speed)
--speed=N%               limit CPU usage in percentage; 100% is one core 
--speed=Nmhz\ghz         limit CPU usage in mhz\ghz
--ncpu=N                 limit VCPU usage (deprecated)
--io=N                   define io limits (KB/s)
--nproc=N                limit number of processes
--pmem=N                 limit physical memory usage for aplications inside LVE
--iops=N                 limit io per second
--mem=N                  mem alias for vmem (deprecated)
--vmem=N                 limit virtual memory for applications inside LVE
--maxEntryProcs=N        limit number of entry processes
--save                   save configuration settings (use with set) (deprecated)
--save-all-parameters    save all parameters even if they match with defaults settings
--json                   returns result of command json formatted
--unlimited              set all limits to unlimited
--save-username          save username in the config file. This parameter is used in conjunction with set-user

Official CloudLinux IO Limit Documentation

lveps

To see if a CloudLinux user has hit their EntryProcs limit you can use this command to view real-time TOP like usage. If you just want to print the current stats once instead of using a top like view remove -t and -c 1 and it will only print out the stats once

lveps -t -c 1

If you just want to view all processes and threads for a specific CloudLinux LVE you can run this command which will give you a much better idea of what is going on for a specific LVE. If the user is having issues loading their website or performing other actions you will want to use this command to see if they reach their EP limit.

lveps -p -n $LVE

##Example Output

      ID     EP    PNO        COM    TNO    TID       CPU      MEM         DT         DO
     LVE      2      2        ---      2    ---       481        0      98944        848
            ---    ---      httpd    ---  17887         1       13        N/A        N/A
            ---    ---      httpd    ---  17864         1       13        N/A        N/A

If you want to list all CloudLinux LVE users sorted by the highest CPU user first, use the -s option with lveps

lveps -p -s cpu

You can also use -s EP to sort users based on the highest EntryProcesses. Basically you can use any limit with -s to sort results however you want.