This box has a single numa node. Each numa node is an entry in the kernel linked list pgdat_list. Each node is further divided into zones. Here are some example zone types:
- DMA Zone: Lower 16 MiB of RAM used by legacy devices that cannot address anything beyond the first 16MiB of RAM.
- DMA32 Zone (only on x86_64): Some devices can't address beyond the first 4GiB of RAM. On x86, this zone would probably be covered by Normal zone
- Normal Zone: Anything above zone DMA and doesn't require kernel tricks to be addressable. Typically on x86, this is 16MiB to 896MiB. Many kernel operations require that the memory being used be from this zone
- Highmem Zone (x86 only): Anything above 896MiB.
Say we have just rebooted the machine and we have a free pool of 16MiB (DMA zone). The most sensible thing to do would be to have the this memory split into largest contiguous blocks available. The largest order is defined at compile time to 11 which means that the largest slice the buddy allocator has is 4MiB block (2^10 * page_size). so the 16 MiB DMA zone would initially split into 4 free blocks.
Here's how we'll service an allocation request for 72KiB:
- Round up the allocation request to the next power of 2 (128)
- Split a 4MiB chunk into two 2MiB chunks
- Split one 2 MiB chunk into two MiB chunks
- Continue splitting until we get a 128KiB chunk that we'll allocate.
Here's an example of an allocation failure from a Gentoo bug report.
In such cases, the buddyinfo proc file will allow you to view the current fragmentation state of your memory.
Here's a quick python script that will make this data more digestible.
And sample output for the buddyinfo data pasted earlier on.
Hi,
ReplyDeleteHow to tackle this issue?
Great article, thanks for sharing some code! I rewrite the code to be able to execute it under 2.4 Python interpreter (that's what you got when with old "enterprise"-like Linux). The code is available here:
ReplyDeletehttps://www.dropbox.com/s/n7n8ru4d4p551zv/buddyinfo.py?dl=0
Sorry, no fancy Github link for now.
I believe that one way to tackle the issue is monitor over time the buddyinfo usage in a server and try to fetch a baseline after the applications are running for some time. Any degradation over the baseline might be an indication that somethings goes wrong (and a schedule server reboot is in order).
I saw that Collectl (http://collectl.sourceforge.net/index.html) might help with that, but I think it is a bit too complicated to figure out how the results are shown.
You are welcome!
DeleteIf we see that the our memory is fragmented, does /proc/buddyinfo actually help? For example, I have a machine with 700 Gig of memory and the "free" command line utility shows that 18 Gig are free. This agrees with the output from your buddyinfo.py. However, I have 573 Gig cached memory. Is not most of that memory available for use, should an application request more memory? So /proc/buddyinfo is cool but it doesn't tell us if our memory is fragmented to the degree that apps are impacted, or not.
ReplyDeleteThanks for this great article!
ReplyDeleteAdded support for the parser arguments - diffs against the original posted above as follows:
ReplyDelete$ diff buddyinfo.py.org buddyinfo.py
7c7,8
< Last author: lmwangi at gmail com
---
> Last author: lmwangi at gmail com
> Parser args support added by: mloeser at cisco com
42c43
< def __init__(self, logger):
---
> def __init__(self, logger, options):
44a46,50
> if options.in_file == None:
> self.in_file = "/proc/buddyinfo"
> else:
> self.in_file = options.in_file
> self.size = options.size
56c62
< buddyinfo = open("/proc/buddyinfo").readlines()
---
> buddyinfo = open(self.in_file).readlines()
89c95,105
< ret_string += " Free KiB in zone: %.2f\n" % (sum(zoneinfo.get("usage")) / (1024.0))
---
> if self.size == "B":
> szstr = "bytes" # Bytes
> div = 1.0
> elif self.size == "M":
> szstr = "mib" # Megabytes
> div = 1024.0 * 1024.0
> else:
> szstr = "kib"
> div = 1024.0 # Kilobytes
> ret_string += " Free %s in zone: %.2f\n" % (szstr,
> sum(zoneinfo.get("usage")) / (div))
91,93c107,108
< "Fragment size", "Free fragments", "Total available KiB",
< width=width,
< align="<")
---
> "Fragment size", "Free fragments", ("Total available %s" % szstr),
> width=width, align="<")
100c115
< usage = zoneinfo.get("usage")[idx] / 1024.0)
---
> usage = zoneinfo.get("usage")[idx] / div)
109a125,126
> parser.add_option("-i", "--input", dest="in_file", action="store", type="string",
> help="Input file")
116c133
< buddy = BuddyInfo(logger)
---
> buddy = BuddyInfo(logger, options)
120c137
< main()
---
> main()
Today, while I was at work, my sister stole my iPad and tested to see if it can survive a forty foot drop, just so she can be a youtube sensation. My apple ipad is now broken and she has 83 views. I know this is entirely off topic but I had to share it with someone!
ReplyDeleteGood info. Lucky me I came across your site by chance (stumbleupon). I've saved it for later!
ReplyDeletepatch for python3
ReplyDelete--- buddyinfo_python2.py
+++ buddyinfo.py
@@ -60,7 +60,7 @@ class BuddyInfo(object):
for line in map(self.parse_line, buddyinfo):
numa_node = int(line["numa_node"])
zone = line["zone"]
- free_fragments = map(int, line["nr_free"].split())
+ free_fragments = list(map(int, line["nr_free"].split()))
max_order = len(free_fragments)
fragment_sizes = self.get_order_sizes(max_order)
usage_in_bytes = [block[0] * block[1] for block in zip(free_fragments, fragment_sizes)]
@@ -115,9 +115,9 @@ def main():
logger = Logger(logging.DEBUG).get_logger()
logger.info("Starting....")
logger.info("Parsed options: %s" % options)
- print logger
+ print(logger)
buddy = BuddyInfo(logger)
- print buddy
+ print(buddy)
if __name__ == '__main__':
main()