Interpreting data from /proc/[id]/mountstats or /proc/self/mountstats

 

I would like to monitor the IOs of my NFS shares. There are plenty of programs which are interpreting the data within /proc/[id]/mountstats ([id] = id of nfs client/server) or /proc/self/mountstats . The problem is, these programs calculate some IO per second. This is very nice in the command line, but it is quite useless for my monitoring solution. So I want to interpret the counters of mountstats myself. My problem now is, that I need some documentation of mountstats for nfs v4.

So far my search haven't brought up any useful.

Thanks for your help

sample output of /proc/self/mountstats on Ubuntu (nfs client): 

...
device IPADRESS:/mountpoint mounted on /mnt/mountpoint with fstype nfs4 statvers=1.0
    opts:   rw,vers=4,rsize=1048576,wsize=1048576,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,nointr,proto=tcp,timeo=600,retrans=2,sec=sys
    age:    77663
    caps:   caps=0x1e,wtmult=512,dtsize=4096,bsize=0,namelen=255
    nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3
    sec:    flavor=1,pseudoflavor=1
    events: 1462723 25756220 1025148 1164028 1083099 2027020 31347107 1984730 279773 1915700 1346583 21077199 520498 31825 525298 39026 2180470 1083041 0 16822 990144 11011 0 0 0
    bytes:  46686144592 3990939437 0 0 30238687174 3759616491 7739053 1346583
    RPC iostats version: 1.0  p/v: 100003/4 (nfs)
    xprt:   tcp 852 1 2 0 0 11822378 11822377 0 17669710 13357205
    per-op statistics
            NULL: 0 0 0 0 0 0 0 0
            READ: 1950114 1950114 0 330348288 30355844804 105050 6678050 6907640
           WRITE: 911822 911822 0 3932410080 215189360 10653830 1472320 12211620
          COMMIT: 389509 389509 0 65718096 88808052 8060 253550 284360
            OPEN: 1168845 1168845 0 292577612 554554444 430040 628900 1229990
    OPEN_CONFIRM: 43 43 0 7048 2924 0 10 10
     OPEN_NOATTR: 0 0 0 0 0 0 0 0
    OPEN_DOWNGRADE: 54 54 0 9952 12744 90 0 120
           CLOSE: 1081161 1081161 0 191495392 255153996 201030 404620 767870
         SETATTR: 43303 43303 0 8753024 10046296 1060 25650 30380
          FSINFO: 2 2 0 268 184 0 0 0
           RENEW: 0 0 0 0 0 0 0 0
     SETCLIENTID: 0 0 0 0 0 0 0 0
    SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
            LOCK: 1140050 1140050 0 246277244 77523400 76270 326950 469890
           LOCKT: 1 1 0 188 52 0 0 0
           LOCKU: 1037895 1037895 0 187916912 70576860 18130 357530 441480
          ACCESS: 450451 450451 0 72511716 102702828 21440 222460 280950
         GETATTR: 1462724 1462724 0 224029496 310097488 21290 390910 485350
          LOOKUP: 2043380 2043380 0 361899452 532433688 18200 499200 585170
     LOOKUP_ROOT: 1 1 0 116 232 0 0 0
          REMOVE: 32356 32356 0 5953448 7765440 600 15450 18560
          RENAME: 20388 20388 0 5593088 9215376 510 7740 9750
            LINK: 537 537 0 132024 231984 0 220 270
         SYMLINK: 0 0 0 0 0 0 0 0
          CREATE: 999 999 0 227376 293140 30 650 830
        PATHCONF: 1 1 0 140 72 0 0 0
          STATFS: 2872 2872 0 419360 333152 580 3980 4880
        READLINK: 0 0 0 0 0 0 0 0
         READDIR: 82718 82718 0 14625720 118955760 770 48470 53310
     SERVER_CAPS: 3 3 0 400 264 0 0 0
     DELEGRETURN: 0 0 0 0 0 0 0 0
          GETACL: 0 0 0 0 0 0 0 0
          SETACL: 0 0 0 0 0 0 0 0
    FS_LOCATIONS: 0 0 0 0 0 0 0 0
 ...

Complete source: http://www.fsl.cs.stonybrook.edu/~mchen/mountstat-format.txt 

       Format of NFS mountstat

Linux kernel version: 2.6.32-358.el6.x86_64

Important files for statistics: 

- linux/fs/nfs/super.c: nfs_show_stats
- linux/include/linux/nfs_iostat.h: nfs_stat_eventcounters
- linux/net/sunrpc/stats.c: rpc_print_iostats
- linux/net/sunrpc/xprtsock.c: xs_tcp_print_stats

Server Capabilities (caps)

  • caps: server capabilities
  • wtmult: server disk block size
  • dtsize: readdir size
  • bsize: server block size
  • namlen:

    • nfsv4 (nfs_show_stats)

  • bm0: attribute_bitmask[0], which attributes are supported
  • bm1: attribute_bitmask[1], which attributes are supported
  • acl: acl_bitmask, which acl attributes are supported

    • sec, security flavor in effect for this mount (nfs_show_stats)

  • flavor: auth->au_ops->au_flavor
  • pseudoflavor: auth->au_flavor

    • Events (linux/nfs_iostat.h: nfs_stat_eventcounters)

  • inoderevalidate
  • dentryrevalidate
  • datainvalidate
  • attrinvalidate
  • vfsopen
  • vfslookup
  • vfsaccess
  • vfsupdatepage
  • vfsreadpage
  • vfsreadpages
  • vfswritepage
  • vfswritepages
  • vfsgetdents
  • vfssetattr
  • vfsflush
  • vfsfsync
  • vfslock
  • vfsrelease
  • congestionwait
  • setattrtrunc
  • extendwrite
  • sillyrename
  • shortread
  • shortwrite
  • delay
  • pnfs_read
  • pnfs_write

    • Bytes (linux/nfs_iostat.h: nfs_stat_bytecounters)

  • normalreadbytes
  • normalwritebytes
  • directreadbytes
  • directwritebytes
  • serverreadbytes
  • serverwritebytes (serverwrittenbytes)
  • readpages
  • writepages

    • xprt (tcp)

  • linux/net/sunrpc/xprtsock.c: xs_tcp_print_stats
  • linux/include/linux/sunrpc/xprt.h: struct rpc_xprt
  • srcport: Ephemeral port
  • bind_count: How many rpcbind operations
  • connect_count: How many TCP connects
  • connect_time: How long connects have taken
  • idle_time: How long transport has been idle
  • rpcsends: How many socket sends
  • rpcrecvs: How many socket receives
  • badxids: How many unmatchable XIDs have been received
  • req_u: Average requests on the wire (slot table utilization)
  • bklog_u: backlog queue utilization (average length of baklog queue)
  • max_slots: max rpc_slots used
  • sending_u: send q utilization
  • pending_u: pend q utilization

    • Each RPC operation type shows (rpc_print_iostats)

    The stats recording: rpc_exit_task ---> xprt_release ---> rpc_count_iostats

    All NFS protocols are explained here: http://www.freesoft.org/CIE/Topics/115.htm

  • The name of the op type
  • ops How many ops of this type have been requested
  • trans: How many transmissions of this op type have been sent
  • timeouts: How many timeouts of this op type have occurred
  • bytes_sent: How many bytes have been sent for this op type
  • bytes_recv: How many bytes have been received for this op type
  • queue: How long ops of this type have waited in queue before being transmitted (microsecond)
  • rtt: How long the client waited to receive replies of this op type from the server (microsecond)
  • execute: How long ops of this type take to execute (from rpc_init_task to rpc_exit_task) (microsecond)

    • rpc_run_task ---> rpc_new_task ---> rpc_init_task

    The last three time statistics are collected using ns, but they are converted to ms when they are printed out.

    The execute time should approximate to (queue + rtt).

    Note, in the Linux source code, the names of the fields are prefixed with "om_", for example, the first field is "om_ops" instead of "ops".

    NFSv4.0 EXAMPLE 

    device 130.245.177.235:/nfsdata/ mounted on /mnt with fstype nfs4 statvers=1.1
    opts:    rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,
            acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,
            clientaddr=130.245.177.234,minorversion=0,local_lock=none
    age:    19  (time, in seconds, since the NFS is mounted)
    caps:    caps=0xfff7,wtmult=512,dtsize=32768,bsize=0,namlen=255
    nfsv4:    bm0=0xfdffafff,bm1=0xf9be3e,acl=0x0
    sec:    flavor=1,pseudoflavor=1
    events:    0 168232 0 0 0 10095 217808 0 2 9797 0 9739 0 0 19739 19739 0 19739 0 0 0 0 0 0 0 0 0
    bytes:    1612840960 0 0 0 627536112 0 158076 0
    RPC iostats version: 1.0  p/v: 100003/4 (nfs)
    xprt:    tcp 737 0 1 0 0 69698 69697 0 81817 0 2 1082 12119
    per-op statistics
            NULL: 0 0 0 0 0 0 0 0
            READ: 9797 9797 0 1567520 628138612 71 7953 8200
           WRITE: 0 0 0 0 0 0 0 0
          COMMIT: 0 0 0 0 0 0 0 0
            OPEN: 19740 19740 0 4737600 7343280 505 3449 4172
    OPEN_CONFIRM: 10211 10211 0 1552072 694348 74 836 1008
     OPEN_NOATTR: 0 0 0 0 0 0 0 0
    OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
           CLOSE: 19739 19739 0 3316152 2605548 334 3045 3620
         SETATTR: 0 0 0 0 0 0 0 0
          FSINFO: 1 1 0 132 108 0 0 0
           RENEW: 0 0 0 0 0 0 0 0
     SETCLIENTID: 0 0 0 0 0 0 0 0
    SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
            LOCK: 0 0 0 0 0 0 0 0
           LOCKT: 0 0 0 0 0 0 0 0
           LOCKU: 0 0 0 0 0 0 0 0
          ACCESS: 96 96 0 14584 19584 0 8 10
         GETATTR: 1 1 0 132 188 0 0 0
          LOOKUP: 10095 10095 0 1655576 2382420 36 898 1072
     LOOKUP_ROOT: 0 0 0 0 0 0 0 0
          REMOVE: 0 0 0 0 0 0 0 0
          RENAME: 0 0 0 0 0 0 0 0
            LINK: 0 0 0 0 0 0 0 0
         SYMLINK: 0 0 0 0 0 0 0 0
          CREATE: 0 0 0 0 0 0 0 0
        PATHCONF: 1 1 0 128 72 0 0 0
          STATFS: 0 0 0 0 0 0 0 0
        READLINK: 0 0 0 0 0 0 0 0
         READDIR: 0 0 0 0 0 0 0 0
     SERVER_CAPS: 2 2 0 256 176 0 0 0
     DELEGRETURN: 0 0 0 0 0 0 0 0
          GETACL: 0 0 0 0 0 0 0 0
          SETACL: 0 0 0 0 0 0 0 0
    FS_LOCATIONS: 0 0 0 0 0 0 0 0
    RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0
         SECINFO: 0 0 0 0 0 0 0 0
     EXCHANGE_ID: 0 0 0 0 0 0 0 0
    CREATE_SESSION: 0 0 0 0 0 0 0 0
    DESTROY_SESSION: 0 0 0 0 0 0 0 0
        SEQUENCE: 0 0 0 0 0 0 0 0
    GET_LEASE_TIME: 0 0 0 0 0 0 0 0
    RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
       LAYOUTGET: 0 0 0 0 0 0 0 0
    GETDEVICEINFO: 0 0 0 0 0 0 0 0
    LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
    LAYOUTRETURN: 0 0 0 0 0 0 0 0


NFSv4.1 EXAMPLE
===============
device 130.245.177.235:/nfsdata mounted on /mnt with fstype nfs4 statvers=1.1
    opts:   rw,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=130.245.177.234,local_lock=none
    age:    44
    impl_id:    name='',domain='',date='0,0'
    caps:   caps=0x3ffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
    nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,bm2=0x800,acl=0x3,sessions,pnfs=not configured
    sec:    flavor=1,pseudoflavor=1
    events: 67379 1692812 0 30749 0 980 1984612 0 0 0 0 0 0 0 96611 96611 0 96611 0 0 0 0 0 0 0 0 0 
    bytes:  0 0 0 0 0 0 0 0 
    RPC iostats version: 1.0  p/v: 100003/4 (nfs)
    xprt:   tcp 674 0 1 0 0 279772 279771 0 649944 0 5 1711 90409
    per-op statistics
            NULL: 0 0 0 0 0 0 0 0
            READ: 0 0 0 0 0 0 0 0
           WRITE: 0 0 0 0 0 0 0 0
          COMMIT: 0 0 0 0 0 0 0 0
            OPEN: 96612 96612 0 27824256 34007424 13349 16253 32061
    OPEN_CONFIRM: 0 0 0 0 0 0 0 0
     OPEN_NOATTR: 0 0 0 0 0 0 0 0
    OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
           CLOSE: 96610 96610 0 19708440 17003360 2936 11082 15386
         SETATTR: 0 0 0 0 0 0 0 0
          FSINFO: 1 1 0 168 152 0 0 0
           RENEW: 0 0 0 0 0 0 0 0
     SETCLIENTID: 0 0 0 0 0 0 0 0
    SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
            LOCK: 0 0 0 0 0 0 0 0
           LOCKT: 0 0 0 0 0 0 0 0
           LOCKU: 0 0 0 0 0 0 0 0
          ACCESS: 16932 16932 0 3183200 2844576 83 1919 2308
         GETATTR: 67380 67380 0 12128380 15632160 336 8367 10644
          LOOKUP: 1236 1236 0 247176 167784 7 153 197
     LOOKUP_ROOT: 0 0 0 0 0 0 0 0
          REMOVE: 0 0 0 0 0 0 0 0
          RENAME: 0 0 0 0 0 0 0 0
            LINK: 0 0 0 0 0 0 0 0
         SYMLINK: 0 0 0 0 0 0 0 0
          CREATE: 980 980 0 219516 305760 5 168 197
        PATHCONF: 1 1 0 164 116 0 0 0
          STATFS: 0 0 0 0 0 0 0 0
        READLINK: 0 0 0 0 0 0 0 0
         READDIR: 0 0 0 0 0 0 0 0
     SERVER_CAPS: 2 2 0 328 280 0 0 0
     DELEGRETURN: 0 0 0 0 0 0 0 0
          GETACL: 0 0 0 0 0 0 0 0
          SETACL: 0 0 0 0 0 0 0 0
    FS_LOCATIONS: 0 0 0 0 0 0 0 0
    RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0
         SECINFO: 0 0 0 0 0 0 0 0
     EXCHANGE_ID: 0 0 0 0 0 0 0 0
    CREATE_SESSION: 0 0 0 0 0 0 0 0
    DESTROY_SESSION: 0 0 0 0 0 0 0 0
        SEQUENCE: 0 0 0 0 0 0 0 0
    GET_LEASE_TIME: 0 0 0 0 0 0 0 0
    RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
       LAYOUTGET: 0 0 0 0 0 0 0 0
    GETDEVICEINFO: 0 0 0 0 0 0 0 0
    LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
    LAYOUTRETURN: 0 0 0 0 0 0 0 0
    SECINFO_NO_NAME: 0 0 0 0 0 0 0 0
    TEST_STATEID: 0 0 0 0 0 0 0 0
    FREE_STATEID: 0 0 0 0 0 0 0 0
    GETDEVICELIST: 0 0 0 0 0 0 0 0
    BIND_CONN_TO_SESSION: 0 0 0 0 0 0 0 0
    DESTROY_CLIENTID: 0 0 0 0 0 0 0 0


# vim: tabstop=2:shiftwidth=2:expandtab:foldmethod=indent

    Also https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/linux/nfs_iostat.h and https://utcc.utoronto.ca/~cks/space/blog/linux/NFSMountstatsIndex for additional reading.

    显然，没有比mountstats实用程序的代码更好的“文档”。

    关于你在做什么Linux，因为对我来说，在Redhat上，mountstats（对于v4）是这样的： 

    [root@riau petri]# cat /proc/1511/mountstats 
device rootfs mounted on / with fstype rootfs
device /proc mounted on /proc with fstype proc
device /sys mounted on /sys with fstype sysfs
device udev mounted on /dev with fstype devtmpfs
device devpts mounted on /dev/pts with fstype devpts
device tmpfs mounted on /dev/shm with fstype tmpfs
device /dev/mapper/vg_riau-lv_root mounted on / with fstype ext4
device /proc/bus/usb mounted on /proc/bus/usb with fstype usbfs
device /dev/sda1 mounted on /boot with fstype ext4
device /dev/mapper/vg_riau-lv_home mounted on /home with fstype ext4
device none mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
device cgroup mounted on /cgroup/cpuset with fstype cgroup
device cgroup mounted on /cgroup/cpu with fstype cgroup
device cgroup mounted on /cgroup/cpuacct with fstype cgroup
device cgroup mounted on /cgroup/memory with fstype cgroup
device cgroup mounted on /cgroup/devices with fstype cgroup
device cgroup mounted on /cgroup/freezer with fstype cgroup
device cgroup mounted on /cgroup/net_cls with fstype cgroup
device cgroup mounted on /cgroup/blkio with fstype cgroup
device sunrpc mounted on /var/lib/nfs/rpc_pipefs with fstype rpc_pipefs
device /etc/auto.misc mounted on /misc with fstype autofs
device -hosts mounted on /net with fstype autofs
device nfsd mounted on /proc/fs/nfsd with fstype nfsd
    链接地址: http://www.djcxy.com/p/58990.html

    上一篇: 如何模拟从Controller测试类中的域类中注入的服务？

    下一篇: 解释/ proc / [id] / mountstats或/ proc / self / mountstats中的数据