Pages

Tuesday, August 25, 2015

How To Use Find and Locate to Search for Files on a Linux VPS

Sep 27, 2013  Linux BasicsSystem ToolsLinux Commands

Introduction

One problem users run into when first dealing with a Linux machine is how to find the files they are looking for.
This guide will cover how to use the aptly named find command. This will help you search for files on your VPS using a variety of filters and parameters. We will also briefly cover the locate command, which can be used to search for commands in a different way.

Finding by Name

The most obvious way of searching for files is by name.
To find a file by name, type:
find -name "query"
This will be case sensitive, meaning a search for "file" is different than a search for "File".
To find a file by name, but ignore the case of the query, type:
find -iname "query"
If you want to find all files that don't adhere to a specific pattern, you can invert the search with "-not" or "!". If you use "!", you must escape the character so that bash does not try to interpret it before find can act:
find -not -name "query_to_avoid"
Or
find \! -name "query_to_avoid"

Finding by Type

You can specify the type of files you want to find with the "-type" parameter. It works like this:
find -type type_descriptor query
Some of the most common descriptors that you can use to specify the type of file are here:
  • f: regular file
  • d: directory
  • l: symbolic link
  • c: character devices
  • b: block devices
For instance, if we wanted to find all of the character devices on our system, we could issue this command:
find / -type c
/dev/parport0
/dev/snd/seq
/dev/snd/timer
/dev/autofs
/dev/cpu/microcode
/dev/vcsa7
/dev/vcs7
/dev/vcsa6
/dev/vcs6
/dev/vcsa5
/dev/vcs5
/dev/vcsa4
. . .
We can search for all files that end in ".conf" like this:
find / -type f -name "*.conf"
/var/lib/ucf/cache/:etc:rsyslog.d:50-default.conf
/usr/share/base-files/nsswitch.conf
/usr/share/initramfs-tools/event-driven/upstart-jobs/mountall.conf
/usr/share/rsyslog/50-default.conf
/usr/share/adduser/adduser.conf
/usr/share/davfs2/davfs2.conf
/usr/share/debconf/debconf.conf
/usr/share/doc/apt-utils/examples/apt-ftparchive.conf
. . .

Filtering by Time and Size

Find gives you a variety of ways to filter results by size and time.

Size

You can filter by size with the use of the "-size" parameter.
We add a suffix on the end of our value that specifies how we are counting. These are some popular options:
  • c: bytes
  • k: Kilobytes
  • M: Megabytes
  • G: Gigabytes
  • b: 512-byte blocks
To find all files that are exactly 50 bytes, type:
find / -size 50c
To find all files less than 50 bytes, we can use this form instead:
find / -size -50c
To Find all files more than 700 Megabytes, we can use this command:
find / -size +700M

Time

Linux stores time data about access times, modification times, and change times.
  • Access Time: Last time a file was read or written to.
  • Modification Time: Last time the contents of the file were modified.
  • Change Time: Last time the file's inode meta-data was changed.
We can use these with the "-atime", "-mtime", and "-ctime" parameters. These can use the plus and minus symbols to specify greater than or less than, like we did with size.
The value of this parameter specifies how many days ago you'd like to search.
To find files that have a modification time of a day ago, type:
find / -mtime 1
If we want files that were accessed in less than a day ago, we can type:
find / -atime -1
To get files that last had their meta information changed more than 3 days ago, type:
find / -ctime +3
There are also some companion parameters we can use to specify minutes instead of days:
find / -mmin -1
This will give the files that have been modified type the system in the last minute.
Find can also do comparisons against a reference file and return those that are newer:
find / -newer myfile

Finding by Owner and Permissions

You can also search for files by the file owner or group owner.
You do this by using the "-user" and "-group" parameters respectively. Find a file that is owned by the "syslog" user by entering:
find / -user syslog
Similarly, we can specify files owned by the "shadow" group by typing:
find / -group shadow
We can also search for files with specific permissions.
If we want to match an exact set of permissions, we use this form:
find / -perm 644
This will match files with exactly the permissions specified.
If we want to specify anything with at least those permissions, you can use this form:
find / -perm -644
This will match any files that have additional permissions. A file with permissions of "744" would be matched in this instance.

Filtering by Depth

For this section, we will create a directory structure in a temporary directory. It will contain three levels of directories, with ten directories at the first level. Each directory (including the temp directory) will contain ten files and ten subdirectories.
Make this structure by issuing the following commands:
cd
mkdir -p ~/test/level1dir{1..10}/level2dir{1..10}/level3dir{1..10}
touch ~/test/{file{1..10},level1dir{1..10}/{file{1..10},level2dir{1..10}/{file{1..10},level3dir{1..10}/file{1..10}}}}
cd ~/test
Feel free to check out the directory structures with ls and cd to get a handle on how things are organized. When you are finished, return to the test directory:
cd ~/test
We will work on how to return specific files from this structure. Let's try an example with just a regular name search first, for comparison:
find -name file1
./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
./level1dir7/level2dir8/level3dir6/file1
./level1dir7/level2dir8/level3dir5/file1
./level1dir7/level2dir8/file1
. . .
There are a lot of results. If we pipe the output into a counter, we can see that there are 1111 total results:
find -name file1 | wc -l
1111
This is probably too many results to be useful to you in most circumstances. Let's try to narrow it down.
You can specify the maximum depth of the search under the top-level search directory:
find -maxdepth num -name query
To find "file1" only in the "level1" directories and above, you can specify a max depth of 2 (1 for the top-level directory, and 1 for the level1 directories):
find -maxdepth 2 -name file1
./level1dir7/file1
./level1dir1/file1
./level1dir3/file1
./level1dir8/file1
./level1dir6/file1
./file1
./level1dir2/file1
./level1dir9/file1
./level1dir4/file1
./level1dir5/file1
./level1dir10/file1
That is a much more manageable list.
You can also specify a minimum directory if you know that all of the files exist past a certain point under the current directory:
find -mindepth num -name query
We can use this to find only the files at the end of the directory branches:
find -mindepth 4 -name file
./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
. . .
Again, because of our branching directory structure, this will return a large number of results (1000).
You can combine the min and max depth parameters to focus in on a narrow range:
find -mindepth 2 -maxdepth 3 -name file
./level1dir7/level2dir8/file1
./level1dir7/level2dir5/file1
./level1dir7/level2dir7/file1
./level1dir7/level2dir2/file1
./level1dir7/level2dir10/file1
./level1dir7/level2dir6/file1
./level1dir7/level2dir3/file1
./level1dir7/level2dir4/file1
./level1dir7/file1
. . .

Executing and Combining Find Commands

You can execute an arbitrary helper command on everything that find matches by using the "-exec" parameter. This is called like this:
find find_parameters -exec command_and_params {} \;
The "{}" is used as a placeholder for the files that find matches. The "\;" is used so that find knows where the command ends.
For instance, we could find the files in the previous section that had "644" permissions and modify them to have "664" permissions:
cd ~/test
find . -type f -perm 644 -exec chmod 664 {} \;
We could then change the directory permissions like this:
find . -type d -perm 755 -exec chmod 700 {} \;
If you want to chain different results together, you can use the "-and" or "-or" commands. The "-and" is assumed if omitted.
find . -name file1 -or -name file9 

Find Files Using Locate

An alternative to using find is the locate command. This command is often quicker and can search the entire file system with ease.
You can install the command with apt-get:
sudo apt-get update
sudo apt-get install mlocate
The reason locate is faster than find is because it relies on a database of the files on the filesystem.
The database is usually updated once a day with a cron script, but you can update it manually by typing:
sudo updatedb
Run this command now. Remember, the database must always be up-to-date if you want to find recently acquired or created files.
To find files with locate, simply use this syntax:
locate query
You can filter the output in some ways.
For instance, to only return files containing the query itself, instead of returning every file that has the query in the directories leading to it, you can use the "-b" for only searching the "basename":
locate -b query
To have locate only return results that still exist in the filesystem (that were not remove between the last "updatedb" call and the current "locate" call), use the "-e" flag:
locate -e query
To see statistics about the information that locate has cataloged, use the "-S" option:
locate -S
Database /var/lib/mlocate/mlocate.db:
    3,315 directories
    37,228 files
    1,504,439 bytes in file names
    594,851 bytes used to store database

Conclusion

Both find and locate are good ways to find files on your system. It is up to you to decide which of these tools is appropriate in each situation.
Find and locate are powerful commands that can be strengthened by combining them with other utilities through pipelines. Experiment with filtering by using commands like wcsort, and grep.
By Justin Ellingwood

Friday, August 21, 2015

HugePages

HugePages是通过使用大页内存来取代传统的4kb内存页面,使得管理虚拟地址数变少,加快了从虚拟地址到物理地址的映射以及通过摒弃内存页面的换入换出以提高内存的整体性能。尤其是对于8GB以上的内存以及较大的Oracle SGA size,建议配值并使用HugePage特性。本文基于x86_64 Linux下来描述如何配值 HugePages。
    有关HugePages的特性请参考:Linux HugePage 特性
 
1、为什么需要配值HugePages ?a、Larger Page Size and Less # of Pages:     Default page size is 4K whereas the HugeTLB size is 2048K. That means the system would need to handle 512 times less pages.

b、No Page Table Lookups:     Since the HugePages are not subject to replacement (despite regular pages), page table lookups are not required.

c、Better Overall Memory Performance:     On virtual memory systems (any modern OS) each memory operation is actually two abstract memory operations. With HugePages, since there are less number of pages to work on, the possible bottleneck on page table access is clearly avoided.

d、No Swapping:
    We must avoid swapping to happen on Linux OS at all Document 1295478.1. HugePages are not swappable (whereas regular pages are). Therefore there is no page replacement mechanism overhead. HugePages are universally regarded as pinned.

e、No 'kswapd' Operations:     kswapd will get very busy if there is a very large area to be paged (i.e. 13 million page table entries for 50GB memory) and will use an incredible amount of CPU resource. When HugePages are used, kswapd is not involved in managing them. See also Document 361670.1

2、配值HugePages  下面列出了配值HugePages的所有步骤
a、查看当前系统是否配值HugePages  下面的查询中HugePages相关的几个值都为0,表明当前未配值HugePages,其次可以看到Hugepagesize为2MB。
  $ grep Huge /proc/meminfo
  HugePages_Total:   0
  HugePages_Free:    0
  HugePages_Rsvd:    0
  Hugepagesize:     2048 kB
   
b、修改用户的memlock限制  通过修改/etc/security/limits.conf 配值文件来实现
  该参数的值通常配值位略小于当前的已安装系统内存,如当前你的系统内存为64GB,可以做如下设置
  *   soft   memlock    60397977
  *   hard   memlock    60397977
  上述的设置单位为kb,不会降低系统性能。至少也要配值为略大于系统上所有SGA的总和。
  使用ulimit -l 来校验该设置

c、禁用AMM(Oracle 11g)  如果当前的Oracle 版本为10g,可以跳过此步骤。
  如果当前的Oracle 版本为11g,由于AMM(Automatic Memory Management)特性与Hugepages不兼容,需要禁用AMM。
    ALTER SYSTEM RESET memory_target SCOPE=SPFILE;
    ALTER SYSTEM RESET memory_max_target SCOPE=SPFILE;
    ALTER SYSTEM SET sga_target=<n>g SCOPE=SPFILE;
    ALTER SYSTEM SET pga_aggregate_target=<n>g SCOPE=SPFILE;
    SHUTDOWN IMMEDIATE; 
    STARTUP;
   
d、计算vm.nr_hugepages 的值   
  使用Oracle 提供的脚本hugepages_settings.sh的脚本来计算vm.nr_hugepages的值
  在执行脚本之前确保所有的Oracle 实例已启动以及ASM也启动(存在的情形下)
  $ ./hugepages_settings.sh
  ...
  Recommended setting: vm.nr_hugepages = 1496

e、 编辑/etc/sysctl.conf 来设置vm.nr_hugepages参数  $ sysctl -w vm.nr_hugepages = 1496 
  $ sysctl -p
 
  -- Author : Robinson
  -- Blog   : 
http://blog.csdn.net/robinson_0612
 
f、停止所有的Instance并重启server  上述的所有步骤已经实现了动态修改,但对于HugePages的分配需要重新启动server才能生效。

h、验证配值  HugePages相关参数的值会随着当前服务器上的实例的停止与启动而动态发生变化
  通常情况下,HugePages_Free的值应当小于HugePages_Total的值,在HugePages被使用时HugePages_Rsvd值应当为非零值。
  $ grep Huge /proc/meminfo
  HugePages_Total:   131
  HugePages_Free:     20
  HugePages_Rsvd:     20
  Hugepagesize:     2048 kB
 
  如下面的情形,当服务器上仅有的一个实例被关闭后,HugePages_Rsvd的值为零。且HugePages_Free等于HugePages_Total
  $ grep Huge /proc/meminfo
  HugePages_Total:   131
  HugePages_Free:    131
  HugePages_Rsvd:      0
  Hugepagesize:     2048 kB   

3、使用HugePages的注意事项  下面的三种情形应当重新配置HugePages
    a、物理内存的增减或减少
    b、在当前服务器上新增或移出Instance
    c、Instance的SGA大小增加或减少  
  如果未能调整HugePages,可能会引发下面的问题
    a、数据库性能地下
    b、出现内存不足或者过度使用交换空间
    c、数据库实例不能被启动
    d、关键性系统服务故障
  
4、HugePages特性的常见故障处理
Symptom A:    System is running out of memory or swapping
Possible Cause:
    Not enough HugePages to cover the SGA(s) and therefore the area reserved for HugePages are wasted where SGAs are allocated through regular pages.
Troubleshooting Action:
    Review your HugePages configuration to make sure that all SGA(s) are covered.

Symptom B:    Databases fail to start
Possible Cause:
    memlock limits are not set properly
Troubleshooting Action:
    Make sure the settings in limits.conf apply to database owner account.

Symptom C:    One of the database fail to start while another is up
Possible Cause:
    The SGA of the specific database could not find available HugePages and remaining RAM is not enough.
Troubleshooting Action:
    Make sure that the RAM and HugePages are enough to cover all your database SGAs

Symptom D:    Cluster Ready Services (CRS) fail to start
Possible Cause:
    HugePages configured too large (maybe larger than installed RAM)
Troubleshooting Action:
    Make sure the total SGA is less than the installed RAM and re-calculate HugePages.

Symptom E:    HugePages_Total = HugePages_Free
Possible Cause:
    HugePages are not used at all. No database instances are up or using AMM.
Troubleshooting Action:
   Disable AMM and make sure that the database instances are up.

Symptom F:    Database started successfully and the performance is slow
Possible Cause:
    The SGA of the specific database could not find available HugePages and therefore the SGA is handled by regular pages, which leads to slow performance
Troubleshooting Action:
    Make sure that the HugePages are many enough to cover all your database SGAs
Reference: [ID 361468.1]

5、计算vm.nr_hugepages 值的脚本

  1. #!/bin/bash  
  2. #  
  3. # hugepages_settings.sh  
  4. #  
  5. # Linux bash script to compute values for the  
  6. # recommended HugePages/HugeTLB configuration  
  7. #  
  8. # Note: This script does calculation for all shared memory  
  9. # segments available when the script is run, no matter it  
  10. # is an Oracle RDBMS shared memory segment or not.  
  11. #  
  12. # This script is provided by Doc ID 401749.1 from My Oracle Support   
  13. # http://support.oracle.com  
  14.   
  15. # Welcome text  
  16. echo "  
  17. This script is provided by Doc ID 401749.1 from My Oracle Support   
  18. (http://support.oracle.com) where it is intended to compute values for   
  19. the recommended HugePages/HugeTLB configuration for the current shared   
  20. memory segments. Before proceeding with the execution please note following:  
  21.  * For ASM instance, it needs to configure ASMM instead of AMM.  
  22.  * The 'pga_aggregate_target' is outside the SGA and   
  23.    you should accommodate this while calculating SGA size.  
  24.  * In case you changes the DB SGA size,   
  25.    as the new SGA will not fit in the previous HugePages configuration,   
  26.    it had better disable the whole HugePages,   
  27.    start the DB with new SGA size and run the script again.  
  28. And make sure that:  
  29.  * Oracle Database instance(s) are up and running  
  30.  * Oracle Database 11g Automatic Memory Management (AMM) is not setup   
  31.    (See Doc ID 749851.1)  
  32.  * The shared memory segments can be listed by command:  
  33.      # ipcs -m  
  34.   
  35. Press Enter to proceed..."  
  36.   
  37. read  
  38.   
  39. # Check for the kernel version  
  40. KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`  
  41.   
  42. # Find out the HugePage size  
  43. HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`  
  44. if [ -z "$HPG_SZ" ];then  
  45.     echo "The hugepages may not be supported in the system where the script is being executed."  
  46.     exit 1  
  47. fi  
  48.   
  49. # Initialize the counter  
  50. NUM_PG=0  
  51.   
  52. # Cumulative number of pages required to handle the running shared memory segments  
  53. for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`  
  54. do  
  55.     MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`  
  56.     if [ $MIN_PG -gt 0 ]; then  
  57.         NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`  
  58.     fi  
  59. done  
  60.   
  61. RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`  
  62.   
  63. # An SGA less than 100MB does not make sense  
  64. # Bail out if that is the case  
  65. if [ $RES_BYTES -lt 100000000 ]; then  
  66.     echo "***********"  
  67.     echo "** ERROR **"  
  68.     echo "***********"  
  69.     echo "Sorry! There are not enough total of shared memory segments allocated for   
  70. HugePages configuration. HugePages can only be used for shared memory segments   
  71. that you can list by command:  
  72.   
  73.     # ipcs -m  
  74.   
  75. of a size that can match an Oracle Database SGA. Please make sure that:  
  76.  * Oracle Database instance is up and running   
  77.  * Oracle Database 11g Automatic Memory Management (AMM) is not configured"  
  78.     exit 1  
  79. fi  
  80.   
  81. # Finish with results  
  82. case $KERN in  
  83.     '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;  
  84.            echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;  
  85.     '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;  
  86.      *) echo "Unrecognized kernel version $KERN. Exiting." ;;  
  87. esac  
  88.   
  89. # End