现在的位置: 首页 > IT运维 > 正文

linux iowait过高的分析

2011年04月26日 IT运维 ⁄ 共 4318字 暂无评论 ⁄ 被围观 298+

linux下查找最耗iowait的进程
2010-07-08 13:21
抓哪个进程干坏事前要先停掉syslog

service syslog stop

打开block dump:

echo 1 > /proc/sys/vm/block_dump

统计:

dmesg | egrep "READ|WRITE|dirtied" | egrep -o \'([a-zA-Z]*)\' | sort | uniq -c | sort -rn | head
1423 kjournald
1075 pdflush
209 indexer
3 cronolog
1 rnald
1 mysqld

不要忘记在抓完之 后关掉block_dump和启动syslog:

echo 0 > /proc/sys/vm/block_dump
service syslog start

加个有人写了一个perl脚本来处理输出,能得到更直观的结果:

参考:http://www.xaprb.com/blog/2009/08/23/how-to-find-per-process-io-statistics-on-linux/

wget http://aspersa.googlecode.com/svn/trunk/iodump
echo 1 > /proc/sys/vm/block_dump
for i in {1..300}; do sleep 1; dmesg -c; done | perl iodump
echo 0 > /proc/sys/vm/block_dump

while true; do sleep 1; dmesg -c; done | perl iodump

TASK                   PID      TOTAL       READ      WRITE      DIRTY DEVICES
kjournald             2034       5672          0       5672          0 sda8
pdflush                300       1164          0       1164          0 sda2, sda5, sda8, sda7, sda3
kjournald              677        639          0        639          0 sda5
kjournald             2038        355          0        355          0 sda3
kjournald             2040        178          0        178          0 sda2
syslogd              29669         43          0         43          0 sda2
kjournald             2036         35          0         35          0 sda7
php-cgi               8337         11         11          0          0 sda8
kswapd0                301          6          0          6          0 sda5, sda8, sda3
php-cgi               8352          6          6          0          0 sda8
php-cgi               8383          4          4          0          0 sda8
php-cgi               8346          3          3          0          0 sda8
nginx                28097          3          3          0          0 sda8
nginx                28100          3          3          0          0 sda8, sda3
php-cgi               8369          3          3          0          0 sda8
php-cgi               8373          2          2          0          0 sda8
php-cgi               8350          2          2          0          0 sda8
nginx                28099          2          2          0          0 sda8
php-cgi               8374          1          1          0          0 sda8
perl                  9185          1          1          0          0 sda5

注意:这里的pid是线程id,需要用ps -eLf|grep pid查询相关的进程。
很高的%wio,即系统I/O繁忙,进程获得cpu但在等I/O,这部分时间的比例,有可能是I/O存在问题??……
iowait太高的话,一般是磁盘成为了瓶颈,因为ide和sata硬盘在读写的时候都要占用cpu……
可以用下面的命令将 cpu 占用率高的线程找出来:
ps H -eo user,pid,ppid,tid,time,%cpu,cmd --sort=%cpu
首先指定参数\'H\',显示线程相关的信息,格式输出中包含:user,pid,ppid,tid,time,%cpu,cmd,然后再用%cpu字段进行排序。
===============

kjournald进程占用了大量CPU,如何优化?
主要运行的是mysql 服务器, 数据操作非常频繁, 发现运行时间比较长以后, kjournald占用了大量的CPU, 导致系统非常缓慢, 有什么优化方式么?
文件系统使用的是ext3

meiki05 发表于 2008-08-21 21:05
调整一下文件系统的jounal模式,默认为ordered ,改成writeback会提高一些效率。

Despite writing some data more than once, ext3 is often faster (higher throughput) than ext2 because ext3\'s journaling optimizes hard drive head motion. You can choose from three journaling modes to optimize speed, optionally choosing to trade off some data integrity.

* One mode, data=writeback, limits the data integrity guarantees, allowing old data to show up in files after a crash, for a potential increase in speed under some circumstances. (This mode, which is the default journaling mode for most journaling file systems, essentially provides the more limited data integrity guarantees of the ext2 file system and merely avoids the long file system check at boot time.)

* The second mode, data=ordered (the default mode), guarantees that the data is consistent with the file system; recently-written files will never show up with garbage contents after a crash.

* The last mode, data=journal, requires a larger journal for reasonable speed in most cases and therefore takes longer to recover in case of unclean shutdown, but is sometimes faster for certain database operations.

The default mode is recommended for general-purpose computing needs. To change the mode, add the data=something option to the mount options for that file system in the /etc/fstab file, as documented in the mount man page (man mount).

meiki05 发表于 2008-08-21 21:29
另外可以用dumpe2fs看一下Journal size是多大,
可以想办法把Journal size的值改大一些。

=============
1.kswapd0

Linux uses kswapd for virtual memory management such that pages that have been recently accessed are kept in memory and less active pages are paged out to disk.
(what is a page?)…Linux uses manages memory in units called pages.
So,the kswapd process regularly decreases the ages of unreferenced pages…and at the end they are paged out(moved out) to disk

系统每过一定时间就会唤醒kswapd,看看内存是否紧张,如果不紧张,则睡眠,在kswapd中,有2个阀值,pages_hige和pages_low,当空闲内存页的数量低于pages_low的时候,kswapd进程就会扫描内存并且每次释放出32个free pages,直到free page的数量到达pages_high.

2.kjournald

EXT3文件系统的日志进程,具有3种模式:

journal - logs all filesystem data and metadata changes. The slowest of the three ext3 journaling modes, this journaling mode minimizes the chance of losing the changes you have made to any file in an ext3 filesystem.(记录所有文件系统上的元数据改变,最慢的一种模式,)

ordered - only logs changes to filesystem metadata, but flushes file data updates to disk before making changes to associated filesystem metadata. This is the default ext3 journaling mode.(默认使用的模式,只记录文件系统改变的元数据,并在改变之前记录日志)

writeback - only logs changes to filesystem metadata but relies on the standard filesystem write process to write file data changes to disk. This is the fastest ext3 journaling mode.(最快的一种模式,同样只记录修改过的元数据,依赖标准文件系统写进程将数据写到硬盘)

修改模式EXT3的工作模式;

vim /etc/fstab

/dev/hda5 /opt ext3 data=writeback 1 0

详细介绍:
http://www.linuxplanet.com/linuxplanet/reports/4136/5/

3.pdflush

pdflush用于将内存中的内容和文件系统进行同步,比如说,当一个文件在内存中进行修改,pdflush负责将它写回硬盘.每当内存中的垃圾页(dirty page)超过10%的时候,pdflush就会将这些页面备份回硬盘.这个比率是可调节的,通过/etc/sysctl.conf中的 vm.dirty_background_ratio项 默认值为10 也可以

cat /proc/sys/vm/dirty_background_ratio 查看当前的值

给我留言

您必须 [ 登录 ] 才能发表留言!

×
#