{"head":"b12defdc619df06fafb50cc7535a919224daa63c","size":1,"push_id":46623717,"ref":"refs/heads/master","commits":[{"sha":"b12defdc619df06fafb50cc7535a919224daa63c","author":{"name":"Matthew Dillon","email":"2b0b4fb77523f35f1f09f723e46229a59cb29d62@apollo.backplane.com"},"url":"https://api.github.com/repos/DragonFlyBSD/DragonFlyBSD/commits/b12defdc619df06fafb50cc7535a919224daa63c","message":"kernel - Major SMP performance patch / VM system, bus-fault/seg-fault fixes\n\nThis is a very large patch which reworks locking in the entire VM subsystem,\nconcentrated on VM objects and the x86-64 pmap code.  These fixes remove\nnearly all the spin lock contention for non-threaded VM faults and narrows\ncontention for threaded VM faults to just the threads sharing the pmap.\n\nMulti-socket many-core machines will see a 30-50% improvement in parallel\nbuild performance (tested on a 48-core opteron), depending on how well\nthe build parallelizes.\n\nAs part of this work a long-standing problem on 64-bit systems where programs\nwould occasionally seg-fault or bus-fault for no reason has been fixed.  The\nproblem was related to races between vm_fault, the vm_object collapse code,\nand the vm_map splitting code.\n\n* Most uses of vm_token have been removed.  All uses of vm_spin have been\n  removed.  These have been replaced with per-object tokens and per-queue\n  (vm_page_queues[]) spin locks.\n\n  Note in particular that since we still have the page coloring code the\n  PQ_FREE and PQ_CACHE queues are actually many queues, individually\n  spin-locked, resulting in very excellent MP page allocation and freeing\n  performance.\n\n* Reworked vm_page_lookup() and vm_object->rb_memq.  All (object,pindex)\n  lookup operations are now covered by the vm_object hold/drop system,\n  which utilize pool tokens on vm_objects.  Calls now require that the\n  VM object be held in order to ensure a stable outcome.\n\n  Also added vm_page_lookup_busy_wait(), vm_page_lookup_busy_try(),\n  vm_page_busy_wait(), vm_page_busy_try(), and other API functions\n  which integrate the PG_BUSY handling.\n\n* Added OBJ_CHAINLOCK.  Most vm_object operations are protected by\n  the vm_object_hold/drop() facility which is token-based.  Certain\n  critical functions which must traverse backing_object chains use\n  a hard-locking flag and lock almost the entire chain as it is traversed\n  to prevent races against object deallocation, collapses, and splits.\n\n  The last object in the chain (typically a vnode) is NOT locked in\n  this manner, so concurrent faults which terminate at the same vnode will\n  still have good performance.  This is important e.g. for parallel compiles\n  which might be running dozens of the same compiler binary concurrently.\n\n* Created a per vm_map token and removed most uses of vmspace_token.\n\n* Removed the mp_lock in sys_execve().  It has not been needed in a while.\n\n* Add kmem_lim_size() which returns approximate available memory (reduced\n  by available KVM), in megabytes.  This is now used to scale up the\n  slab allocator cache and the pipe buffer caches to reduce unnecessary\n  global kmem operations.\n\n* Rewrote vm_page_alloc(), various bits in vm/vm_contig.c, the swapcache\n  scan code, and the pageout scan code.  These routines were rewritten\n  to use the per-queue spin locks.\n\n* Replaced the exponential backoff in the spinlock code with something\n  a bit less complex and cleaned it up.\n\n* Restructured the IPIQ func/arg1/arg2 array for better cache locality.\n  Removed the per-queue ip_npoll and replaced it with a per-cpu gd_npoll,\n  which is used by other cores to determine if they need to issue an\n  actual hardware IPI or not.  This reduces hardware IPI issuance\n  considerably (and the removal of the decontention code reduced it even\n  more).\n\n* Temporarily removed the lwkt thread fairq code and disabled a number of\n  features.  These will be worked back in once we track down some of the\n  remaining performance issues.\n\n  Temproarily removed the lwkt thread resequencer for tokens for the same\n  reason.  This might wind up being permanent.\n\n  Added splz_check()s in a few critical places.\n\n* Increased the number of pool tokens from 1024 to 4001 and went to a\n  prime-number mod algorithm to reduce overlaps.\n\n* Removed the token decontention code.  This was a bit of an eyesore and\n  while it did its job when we had global locks it just gets in the way now\n  that most of the global locks are gone.\n\n  Replaced the decontention code with a fall back which acquires the\n  tokens in sorted order, to guarantee that deadlocks will always be\n  resolved eventually in the scheduler.\n\n* Introduced a simplified spin-for-a-little-while function\n  _lwkt_trytoken_spin() that the token code now uses rather than giving\n  up immediately.\n\n* The vfs_bio subsystem no longer uses vm_token and now uses the\n  vm_object_hold/drop API for buffer cache operations, resulting\n  in very good concurrency.\n\n* Gave the vnode its own spinlock instead of sharing vp->v_lock.lk_spinlock,\n  which fixes a deadlock.\n\n* Adjusted all platform pamp.c's to handle the new main kernel APIs.  The\n  i386 pmap.c is still a bit out of date but should be compatible.\n\n* Completely rewrote very large chunks of the x86-64 pmap.c code.  The\n  critical path no longer needs pmap_spin but pmap_spin itself is still\n  used heavily, particularin the pv_entry handling code.\n\n  A per-pmap token and per-pmap object are now used to serialize pmamp\n  access and vm_page lookup operations when needed.\n\n  The x86-64 pmap.c code now uses only vm_page->crit_count instead of\n  both crit_count and hold_count, which fixes races against other parts of\n  the kernel uses vm_page_hold().\n\n  _pmap_allocpte() mechanics have been completely rewritten to remove\n  potential races.  Much of pmap_enter() and pmap_enter_quick() has also\n  been rewritten.\n\n  Many other changes.\n\n* The following subsystems (and probably more) no longer use the vm_token\n  or vmobj_token in critical paths:\n\n  x The swap_pager now uses the vm_object_hold/drop API instead of vm_token.\n\n  x mmap() and vm_map/vm_mmap in general now use the vm_object_hold/drop API\n    instead of vm_token.\n\n  x vnode_pager\n\n  x zalloc\n\n  x vm_page handling\n\n  x vfs_bio\n\n  x umtx system calls\n\n  x vm_fault and friends\n\n* Minor fixes to fill_kinfo_proc() to deal with process scan panics (ps)\n  revealed by recent global lock removals.\n\n* lockmgr() locks no longer support LK_NOSPINWAIT.  Spin locks are\n  unconditionally acquired.\n\n* Replaced netif/e1000's spinlocks with lockmgr locks.  The spinlocks\n  were not appropriate owing to the large context they were covering.\n\n* Misc atomic ops added"}],"legacy":{"head":"b12defdc619df06fafb50cc7535a919224daa63c","size":1,"push_id":46623717,"shas":[["b12defdc619df06fafb50cc7535a919224daa63c","dillon@apollo.backplane.com","kernel - Major SMP performance patch / VM system, bus-fault/seg-fault fixes\n\nThis is a very large patch which reworks locking in the entire VM subsystem,\nconcentrated on VM objects and the x86-64 pmap code.  These fixes remove\nnearly all the spin lock contention for non-threaded VM faults and narrows\ncontention for threaded VM faults to just the threads sharing the pmap.\n\nMulti-socket many-core machines will see a 30-50% improvement in parallel\nbuild performance (tested on a 48-core opteron), depending on how well\nthe build parallelizes.\n\nAs part of this work a long-standing problem on 64-bit systems where programs\nwould occasionally seg-fault or bus-fault for no reason has been fixed.  The\nproblem was related to races between vm_fault, the vm_object collapse code,\nand the vm_map splitting code.\n\n* Most uses of vm_token have been removed.  All uses of vm_spin have been\n  removed.  These have been replaced with per-object tokens and per-queue\n  (vm_page_queues[]) spin locks.\n\n  Note in particular that since we still have the page coloring code the\n  PQ_FREE and PQ_CACHE queues are actually many queues, individually\n  spin-locked, resulting in very excellent MP page allocation and freeing\n  performance.\n\n* Reworked vm_page_lookup() and vm_object->rb_memq.  All (object,pindex)\n  lookup operations are now covered by the vm_object hold/drop system,\n  which utilize pool tokens on vm_objects.  Calls now require that the\n  VM object be held in order to ensure a stable outcome.\n\n  Also added vm_page_lookup_busy_wait(), vm_page_lookup_busy_try(),\n  vm_page_busy_wait(), vm_page_busy_try(), and other API functions\n  which integrate the PG_BUSY handling.\n\n* Added OBJ_CHAINLOCK.  Most vm_object operations are protected by\n  the vm_object_hold/drop() facility which is token-based.  Certain\n  critical functions which must traverse backing_object chains use\n  a hard-locking flag and lock almost the entire chain as it is traversed\n  to prevent races against object deallocation, collapses, and splits.\n\n  The last object in the chain (typically a vnode) is NOT locked in\n  this manner, so concurrent faults which terminate at the same vnode will\n  still have good performance.  This is important e.g. for parallel compiles\n  which might be running dozens of the same compiler binary concurrently.\n\n* Created a per vm_map token and removed most uses of vmspace_token.\n\n* Removed the mp_lock in sys_execve().  It has not been needed in a while.\n\n* Add kmem_lim_size() which returns approximate available memory (reduced\n  by available KVM), in megabytes.  This is now used to scale up the\n  slab allocator cache and the pipe buffer caches to reduce unnecessary\n  global kmem operations.\n\n* Rewrote vm_page_alloc(), various bits in vm/vm_contig.c, the swapcache\n  scan code, and the pageout scan code.  These routines were rewritten\n  to use the per-queue spin locks.\n\n* Replaced the exponential backoff in the spinlock code with something\n  a bit less complex and cleaned it up.\n\n* Restructured the IPIQ func/arg1/arg2 array for better cache locality.\n  Removed the per-queue ip_npoll and replaced it with a per-cpu gd_npoll,\n  which is used by other cores to determine if they need to issue an\n  actual hardware IPI or not.  This reduces hardware IPI issuance\n  considerably (and the removal of the decontention code reduced it even\n  more).\n\n* Temporarily removed the lwkt thread fairq code and disabled a number of\n  features.  These will be worked back in once we track down some of the\n  remaining performance issues.\n\n  Temproarily removed the lwkt thread resequencer for tokens for the same\n  reason.  This might wind up being permanent.\n\n  Added splz_check()s in a few critical places.\n\n* Increased the number of pool tokens from 1024 to 4001 and went to a\n  prime-number mod algorithm to reduce overlaps.\n\n* Removed the token decontention code.  This was a bit of an eyesore and\n  while it did its job when we had global locks it just gets in the way now\n  that most of the global locks are gone.\n\n  Replaced the decontention code with a fall back which acquires the\n  tokens in sorted order, to guarantee that deadlocks will always be\n  resolved eventually in the scheduler.\n\n* Introduced a simplified spin-for-a-little-while function\n  _lwkt_trytoken_spin() that the token code now uses rather than giving\n  up immediately.\n\n* The vfs_bio subsystem no longer uses vm_token and now uses the\n  vm_object_hold/drop API for buffer cache operations, resulting\n  in very good concurrency.\n\n* Gave the vnode its own spinlock instead of sharing vp->v_lock.lk_spinlock,\n  which fixes a deadlock.\n\n* Adjusted all platform pamp.c's to handle the new main kernel APIs.  The\n  i386 pmap.c is still a bit out of date but should be compatible.\n\n* Completely rewrote very large chunks of the x86-64 pmap.c code.  The\n  critical path no longer needs pmap_spin but pmap_spin itself is still\n  used heavily, particularin the pv_entry handling code.\n\n  A per-pmap token and per-pmap object are now used to serialize pmamp\n  access and vm_page lookup operations when needed.\n\n  The x86-64 pmap.c code now uses only vm_page->crit_count instead of\n  both crit_count and hold_count, which fixes races against other parts of\n  the kernel uses vm_page_hold().\n\n  _pmap_allocpte() mechanics have been completely rewritten to remove\n  potential races.  Much of pmap_enter() and pmap_enter_quick() has also\n  been rewritten.\n\n  Many other changes.\n\n* The following subsystems (and probably more) no longer use the vm_token\n  or vmobj_token in critical paths:\n\n  x The swap_pager now uses the vm_object_hold/drop API instead of vm_token.\n\n  x mmap() and vm_map/vm_mmap in general now use the vm_object_hold/drop API\n    instead of vm_token.\n\n  x vnode_pager\n\n  x zalloc\n\n  x vm_page handling\n\n  x vfs_bio\n\n  x umtx system calls\n\n  x vm_fault and friends\n\n* Minor fixes to fill_kinfo_proc() to deal with process scan panics (ps)\n  revealed by recent global lock removals.\n\n* lockmgr() locks no longer support LK_NOSPINWAIT.  Spin locks are\n  unconditionally acquired.\n\n* Replaced netif/e1000's spinlocks with lockmgr locks.  The spinlocks\n  were not appropriate owing to the large context they were covering.\n\n* Misc atomic ops added","Matthew Dillon"]],"ref":"refs/heads/master"}}