history icon indicating copy to clipboard operation
history copied to clipboard

Published 20 hours ago • verified publisher icon golang-design
→

Metadata

📝 Go: A Documentary | https://golang.design/history

Go: A Documentary

by Changkun Ou <changkun.de> (and many inputs from contributors)

This document collects many interesting (publicly observable) issues, discussions, proposals, CLs, and talks from the Go development process, which intents to offer a comprehensive reference of the Go history.

Table of Contents

  • Go: A Documentary
    • Disclaimer
    • Sources
    • Origin
    • Committers
      • Core Authors
      • Compiler/Runtime Team
      • Library/Tools/Security/Community
      • Group Interviews
    • Timeline
    • Language Design
      • Misc
      • Slice (1.2)
      • Package Management (1.4, 1.5, 1.7)
      • Type alias (1.9)
      • Defer (1.14)
      • Error values (1.13)
      • Channel/Select
      • Generics
    • Compiler Toolchain
      • Compiler
      • Linker
      • Debugger
      • Race Detector
      • Tracer
      • Lock Analysis
      • Builder
      • Modules
      • gopls
      • Testing
    • Runtime Core
      • Scheduler
      • Execution Stack
      • Memory Management
        • Allocator
        • Collector
      • Statistics
      • Memory model
      • ABI
      • Misc
    • Standard Library
      • syscall
      • os, io, io/fs, embed
      • go/*
      • sync
        • Map
        • Pool
        • Mutex, RWMutex
        • Groups
        • atomic
      • time
      • context
      • encoding
      • image, x/image
      • math
      • Mobile
      • misc
    • Unclassified But Relevant Links
    • Fun Facts
    • Acknowledgements
    • License

Disclaimer

  • Most of the texts are written as subjective understanding based on public sources
  • Factual and typo errors may occur. Referring to the original link if some text conflicts to your understanding
  • PRs are very welcome for new content submission, bug fixes, and typo fixes
  • Use Issues for discussions

Back To Top

Sources

There are many sources for digging the documents that relate to Go's historical design, and here are some of the official sources:

Back To Top

Origin

Go is a big project driven by a tiny group of people and the crowd of wisdom from the language user community. Here are some core committers to the project that you might interest in following their excellent work.

Go's origin is attractive without doubts. By listening to the talks held by these people, you could learn more about their oral history and fun stories behind the great work.

For instance, here are some exciting talks regarding the historical feats in Go might be your starting points (as a subjective recommendation): talk/rob2007, talk/rob2009, talk/rob2010b, talk/rob2010d, talk/rob2011a, talk/rob2013a, talk/rob2019, talk/robert2015, talk/russ2014, steve2019b, etc.

Back To Top

Committers

Core Authors

The Go was created by Rob, Robert, and Ken initially because Rob were suffered by the slow C++ compiling time, talked to Robert, and luckily Ken was in the next office. Later, Ian joined the project since he showed huge interests and wrote the gccgo. Rob and Ken are retired. Robert and Ian currently work on adding generics to Go. Russ is also one of the core authors of the project in the early stage. Back then, he was a newcomer at Google, and Rob invited Russ for joining the Go team since he knew Russ from way back because of the Plan 9 project. Russ did many fundamental work for the early Go compiler, runtime, as well as the leap of Go 1.5 bootstrap. Now, Russ is the tech leader of the Go team.

  • Rob Pike. (Robert C. Pike, M. Sc.) Website, Blog, GitHub, Twitter, Reddit. (Retired)
    • Alma mater: University of Toronto (BS), California Institute of Technology
    • talk/rob2007 Advanced Topics in Programming Languages: Concurrency/message passing Newsqueak. May 9, 2007
    • talk/rob2009 The Go Programming Language. Nov 27, 2009.
    • talk/rob2010a Go Programming. Google I/O 2010. May 20, 2010
    • talk/rob2010b Origins of Go Concurrency style. Emerging Languages Camp 2010. July 21, 2010.
    • talk/rob2010c Public Static Void. OSCON 2010. Jul 22, 2010.
    • talk/rob2010d Another Go at Language Design. Aug 27, 2010
    • talk/rob2011a Parallelism and Concurrency in Programming Languages. Feb 17, 2011.
    • talk/rob2011b Google Go: Concurrency, Type System, Memory Management and GC. Feb 25, 2011.
    • talk/rob2011c Lexical Scanning in Go. Aug 30, 2011
    • talk/rob2012a Go Concurrency Patterns. Google I/O 2012. Jul 2, 2012.
    • talk/rob2012b Why Learn Go?. Sep 12, 2012.
    • talk/rob2013a The path to Go 1. Mar 14, 2013.
    • talk/rob2013b Go at Google. Apr 13, 2013.
    • talk/rob2013c Go Programming with Rob Pike and Andrew Gerrand. Aug 14, 2013.
    • talk/rob2013d Concurrency Is Not Parallelism. Oct 20, 2013
    • talk/rob2014a Hello Gophers! Gophercon Opening Keynote. GopherCon 2014. Apr 24, 2014
    • talk/rob2014b Implementing a bignum calculator. Sydney Go meeup. Nov, 19 2014.
    • talk/rob2015a The move from C to Go in the toolchain. GopherFest 2015. Jun 10, 2015.
    • talk/rob2015b Go Proverbs. Gopherfest. Nov 18, 2015
    • talk/rob2015c Simplicity is Complicated. dotGo 2015. Dec 2, 2015
    • talk/rob2016a The Design of the Go Assembler. GopherCon 2016. Aug 18, 2016.
    • talk/rob2016b Stacks of Tokens: A study in interfaces. Sydney Go Meetup. September 19, 2016.
    • talk/rob2017 Upspin. Gopherfest 2017. Jun 22, 2017.
    • talk/rob2018a The History of Unix. Nov 7, 2018.
    • talk/rob2018b Go 2 Draft Specifications. Sydney Golang Meetup. Nov 13, 2018
    • talk/rob2019a Creating the Go programming language with Rob Pike & Robert Griesemer. Sep 10, 2019.
    • talk/rob2019b Rob Pike. A Brief History of Go. Dec 12, 2019.
    • talk/rob2020 A Rob Pike Interview. (Date Unclear) Apr 30, 2020.
    • talk/rob2021 The Go Programming Language and Environment. John Lions Distinguished Lecture, UNSW, May 27, 2021.
  • Robert Griesemer. (Dr. Robert Griesemer) GitHub, Twitter
  • Ken Thompson. (Kenneth Lane Thompson, M. Sc.) (Retired)
    • Alma mater: UC Berkeley
    • talk/ken1982a The UNIX System: Making Computers More Productive. 1982.
    • talk/ken1982b UNIX: Making Computers Easier To Use.
    • talk/ken1982c Ken Thompson and Dennis Ritchie Explain UNIX (Bell Labs).
    • talk/ken1998 Ken Thompson and Dennis Ritchie. National Medal of Technology Awards. 1998.
    • talk/ken2013 Systems Architecture, Design, Engineering, and Verification. Jan 17, 2013.
    • talk/ken2019a The Thompson and Ritchie Story. Feb 18, 2019.
    • talk/ken2019b Brian Kernighan interviews Ken Thompson. VCF East 2019. May 4, 2019.
  • Ian Taylor. (Ian Lance Taylor, B. Sc.) Website, GitHub, Quora
    • Alma mater: Yale University
    • talk/ian2007 GCC: Current Topics and Future Directions. February 27, 2007.
    • talk/ian2018 Transition to Go 2. Gopherpalooza 2018. Oct 24, 2018
    • talk/ian2019a Generics in Go. GopherCon 2019. Aug 27, 2019
  • Russ Cox. (Dr. Russell Stensby Cox) Website, Blog, GitHub, Twitter, Reddit, YouTube

Back To Top

Compiler/Runtime Team

Dmitry is not from the Go team but on the dynamic testing tools team from Google. He wrote the scalable goroutine scheduler, many other performance improvements, synchronization primitives, race detector, and blocking profiler that related to the Go runtime. Austin was an intern at Google who worked on the Go project in the early days while pursuing a Ph. D. Later, he joined the Go team after his academic career and work together with Rick for the Go's concurrent GC. He also worked on the current preemptive scheduler and linker. Now, he is leading the Compiler/Runtime team for Go. Keith and David together focus on the Go's compiler backend, notably the current SSA backend. Michael is a recent newcomer to the Go team, his work mainly in the runtime memory system such as the refactoring of memory allocator and runtime metrics.

  • Dmitry Vyukov. (Дмитрий Вьюков, M. Sc.) Website, GitHub, Twitter
    • Alma mater: Bauman Moscow State Technical University
    • talk/dmitry2014 Writing a functional, reliable and fast web application in Go. Sep 25, 2014.
    • talk/dmitry2015a Chat with Dmitry Vyukov on go-fuzz, golang and IT security. Jul 3, 2015
    • talk/dmitry2015b Go Dynamic Tools. GopherCon 2015. Jul 28, 2015.
    • talk/dmitry2016 Dmitry Vyukov Interview. Jun 1, 2016.
    • talk/dmitry2017 Fuzzing: The New Unit Testing. C++ Russia 2017. May 10, 2017.
    • talk/dmitry2018a Fuzzing: new unit testing. GopherCon Russia. Apr 2, 2018.
    • talk/dmitry2018b Syzbot and the Tale of Thousand Kernel Bugs. Sep 1, 2018.
    • talk/dmitry2019 Go scheduler: Implementing language with lightweight concurrency. Oct 14, 2019.
    • talk/dmitry2020 syzkaller: Adventures in Continuous Coverage-guided Kernel Fuzzing. BlueHat IL 2020. Feb 13, 2020.
  • Austin Clements. (Dr. Austin T. Clements) GitHub, Scholar
    • Alma mater: MIT
    • paper/autin2014 The Scalable Commutativity Rule: Designing Scalable Software for Multicore Processors. Doctor Dissertation. May 21, 2014.
    • talk/austin2020 Pardon the Interruption: Loop Preemption in Go 1.14. Nov 12, 2020.
  • Richard Hudson. (Richard L. Hudson, M. Sc.) GitHub (Retired)
    • Alma mater: University of Massachusetts Amherst
    • paper/rick Research List
    • talk/rick2015a Go GC: Solving the Latency Problem. GopherCon 2015. Jul 8, 2015.
    • talk/rick2015b Rick Hudson on Garbage Collection in Go. Dec 21, 2015.
  • Keith Randall. (Dr. Keith H. Randall.) GitHub
    • Alma mater: MIT
    • paper/1998keith Cilk: Efficient Multithreaded Computing. Doctor Dissertation. May 21, 1998.
    • talk/keith2016 Inside the Map Implementation. GopherCon 2016. Jul 12, 2016.
    • talk/keith2017 Generating Better Machine Code with SSA. GopherCon 2017. Jul 24, 2017.
  • David Chase. (Dr. David Chase) Website, Block, GitHub, Twitter, Scholar
    • Alma mater: Rice University
    • paper/1987david Garbage Collection and Other Optimizations. Doctor Dissertation. Aug 1987.
    • talk/david2017 All About The Go Compiler. Jul 20, 2017.
  • Dan Scales.
  • Michael Knyszek. Website, GitHub
    • talk/michael2020 Evolving the Go Memory Manager's RAM and CPU Efficiency. Nov 11, 2020
  • Than McIntosh. GitHub
  • Cherry Zhang. GitHub

Back To Top

Library/Tools/Release/Security/Community

more people and talks should be added...

Back To Top

Group Interviews

Back To Top

Timeline

A timeline helps you identify the point in time about a document that is reflected in Go versions.

Version Release Expected/Actual Date Days Since Last Release (+Delay)
Go 1 - / 2012.03.28 -
Go 1.1 - / 2013.05.13 440
Go 1.2 - / 2013.12.01 202
Go 1.3 - / 2014.06.18 199
Go 1.4 - / 2014.12.10 175
Go 1.5 2015.07.31 / 2015.08.19 252 (+19)
Go 1.6 2016.01.31 / 2016.02.17 182 (+17)
Go 1.7 2016.07.31 / 2016.08.15 180 (+15)
Go 1.8 2017.01.31 / 2017.02.16 185 (+16)
Go 1.9 2017.07.31 / 2017.08.24 189 (+24)
Go 1.10 2018.01.31 / 2018.02.16 176 (+16)
Go 1.11 2018.07.31 / 2018.08.24 189 (+24)
Go 1.12 2019.01.31 / 2019.02.25 185 (+25)
Go 1.13 2019.07.31 / 2019.09.03 190 (+34)
Go 1.14 2020.01.31 / 2020.02.25 175 (+25)
Go 1.15 2020.07.31 / 2020.08.11 168 (+11)
Go 1.16 2021.01.31 / 2021.02.16 189 (+16)
Go 1.17 2021.07.31 / 2021.08.16 181 (+16)
Go 1.18 2022.01.31 / 2022.03.15 196 (+43)

The historical release notes may helpful for general information:

Back To Top

Language Design

Misc

Back To Top

Slice (1.2)

Back To Top

Package Management (1.4, 1.5, 1.7)

Back To Top

Type alias (1.9)

  • design/type-alias Russ Cox, Robert Griesemer. Proposal: Type Aliases. December 16, 2016
    • talk/type-alias GopherCon 2016 - Lightning Talk: Robert Griesemer - Alias Declarations for Go, A proposal. Oct 9, 2016
    • issue/16339 proposal: Alias declarations for Go
    • issue/18130 all: support gradual code repair while moving a type between packages
  • talk/refactor-video Russ Cox. Codebase Refactoring (with help from Go). GothamGo 2016. November 18, 2016.
    • doc/refactor Russ Cox. Codebase Refactoring (with help from Go).

Back To Top

Defer (1.14)

  • design/open-defer Dan Scales, Keith Randall, and Austin Clements. Proposal: Low-cost defers through inline code, and extra funcdata to manage the panic case. 2019-09-23
  • Unsolved defer recover() edge cases.
    • issue/10458 spec: panicking corner-case semantics
    • issue/23531 spec: recover() in nested defer
    • issue/26275 runtime: document the behaviour of Caller and Callers when used in deferred functions
    • issue/34530 spec: clarify when calling recover stops a panic
    • cl/189377 spec: specify the behavior of recover and recursive panics in more detail

Back To Top

Error values (1.13)

  • doc/err2011 Andrew Gerrand. Error handling in Go. July 2011.
  • doc/err-values Rob Pike. Errors are values. January 2015.
  • doc/err-philosophy Dave Cheney. My philosophy for error handling. April 2016.
  • doc/err-gracefully Dave Cheney. Don’t just check errors, handle them gracefully. April 2016.
  • doc/err-stacktrace Dave Cheney. Stack traces and the errors package. June, 12 2016.
  • doc/err-upspin Rob Pike. Error handling in Upspin. December 06, 2017.
  • doc/err-work Damien Neil and Jonathan Amsterdam. Working with Errors in Go 1.13. October 17, 2019.
  • design/err-handling-overview Russ Cox. Error Handling — Problem Overview.
  • doc/err-value-faq Jonathan Amsterdam and Bryan C. Mills. Error Values: Frequently Asked Questions. August 2019.
  • design/err-handle-check Marcel van Lohuizen. Error Handling — Draft Design. August 27, 2018.
  • design/err-try Robert Griesemer. Proposal: A built-in Go error check function, "try". Jun 12, 2019.
    • issue/32437 Proposal: A built-in Go error check function, "try". Decision response.
  • design/err-inspect Jonathan Amsterdam, Damien Neil. Error Inspection — Draft Design. August 27, 2018.
  • design/err-values-overview Russ Cox. Error Values — Problem Overview. August 27, 2018.
  • design/error-values Jonathan Amsterdam, Russ Cox, Marcel van Lohuizen, Damien Neil. Proposal: Go 2 Error Inspection. January 25, 2019
    • issue/29934 Jonathan Amsterdam. proposal: Go 2 error values. Jan 25, 2019.
    • issue/29934-decision Damien Neil. Go 1.13 lunch decision about error values. May 6, 2019.
    • issue/29934-russ Russ Cox. Response, Response regarding "proposal: Go 2 error values". May 7, 2019.
  • design/err-print Marcel van Lohuizen. Error Printing — Draft Design. August 27, 2018.
  • issue/40432 language: Go 2: error handling meta issue
  • issue/40776 proposal: dynamic ignored error detector
  • issue/41198 proposal: errors: add ErrUnimplemented as standard way for interface method to fail.
  • issue/47811 proposal: errors: add Errors as a standard way to represent multiple errors as a single error
  • issue/53435 proposal: wrapping multiple errors

Back To Top

Channel/Select

  • design/lockfree-channels Dmitry Vyukov. Go channels on steroids. Jan 28, 2014
  • issue/8898 runtime: special case timer channels
  • issue/9120 runtime: remove implementation restriction on channel element size
  • issue/37196 time: make Timer/Ticker channels not receivable with old values after Stop or Reset returns
  • issue/8903 runtime: make chan-based generators faster.
  • issue/21806 fairness in select statement.
  • issue/40410 runtime: trim unnecessary fields from scase
  • issue/40641 runtime: race between stack shrinking and channel send/recv leads to bad sudog values
  • issue/37350 reflect: Select panics if array length greater than 1<<16

Back To Top

Generics

Back To Top

Iterator

  • issue/24282 proposal: Go 2: improve for-loop ergonomics
  • issue/40605 proposal: Go 2: iterators
  • issue/43557 proposal: Go 2: function values as iterators
  • issue/47707 proposal: Go 2: spec: for range with defined types
  • issue/48567 proposal: use channels as iterators
  • issue/50112 proposal: package collection and iterator design for go
  • issue/54047 proposal: Go 2: add a new iterator syntax, package, interfaces
  • discuss/54245 discussion: standard iterator interface

Back To Top

Compiler Toolchain

Compiler

  • code/gc0initial Ken Thompson. Go0 compiler initial version. 28 Mar 2008.
  • code/6g Rob Pike. the first complete Go0 compiler. 4 Jun 2008.
  • design/go12symtab Russ Cox. Go 1.2 Runtime Symbol Information. July 2013.
  • design/go13compiler Russ Cox. Go 1.3+ Compiler Overhaul. December 2013
  • design/go14generate Rob Pike. Go generate: A Proposal
  • design/dev.cc Russ Cox. dev.cc branch plan. Nov 2014
  • design/go15bootstrap Russ Cox. Go 1.5 Bootstrap Plan. January 2015.
  • doc/escape-analysis Dmitry Vyukov. Go Escape Analysis Flaws. Feb 10, 2015.
  • design/execmodes Ian Lance Taylor. Go Execution Modes. August, 2014 (updated January 2016)
  • design/go17ssa Keith Randall. New SSA Backend for the Go Compiler. 2/10/2015.
  • doc/compiler-optimization Compiler And Runtime Optimizations.
  • issue/6853 all: binaries too big and growing.
  • design/go19inlining David Lazar, Austin Clements. Proposal: Mid-stack inlining in the Go compiler.
  • design/dwarf-inlining Than McIntosh. Proposal: emit DWARF inlining info in the Go compiler. 2017-10-23.
    • issue/22080 cmd/compile: generated DWARF inlining info
  • issue/23109 cmd/compile: rewrite escape analysis.
  • issue/27167 cmd/compile: rename a bunch of things
  • GOEXPERIMENT=checkptr
  • issue/37121 runtime: unaligned jumps causing performance regression on Intel
  • issue/16798 proposal: cmd/compile: add tail call optimization for self-recursion only. (declined)
  • issue/22624 proposal: Go 2: add become statement to support tail calls (declined)
  • design/64align Dan Scales. Proposal: Make 64-bit fields be 64-bit aligned on 32-bit systems, add //go:packed, //go:align directives. 2020-06-08.
    • issue/599 cmd/compile: make 64-bit fields 64-bit aligned on 32-bit systems
    • issue/36606 proposal: cmd/compile: make 64-bit fields be 64-bit aligned on 32-bit systems, add //go:packed directive on structs
  • talk/gccgo Brief overview of gccgo, "the other" Go compiler. Aug 6, 2015.
  • issue/28262 cmd/compile: feedback-guided optimization

Back To Top

Linker

The Go Linker was written by Ken Thompson. Russ led a few more overhaul in Go 1.3. Austin led a rework to the linker together with Keith, Than, Cheery and many other brilliant brains, which was landed in Go 1.15 and Go 1.16.

Back To Top

Debugger

Back To Top

Race Detector

  • issue/42598 runtime: apparent false-positive race report for a buffered channel after CL 220419

Back To Top

Tracer

Back To Top

Lock Analysis

  • issue/38029 x/build: linux-amd64-staticlockranking consistently failing
    • cl/192704 runtime: lock operation logging support
    • cl/207348 runtime: release timersLock while running timer
    • cl/207846 runtime: avoid potential deadlock when tracing memory code
    • cl/207619 runtime: static lock ranking for the runtime (enabled by GOEXPERIMENT)
    • cl/222925 cmd/go: define a build tag for any GOEXPERIMENT which is enabled
    • cl/228417 runtime: incorporate Gscan acquire/release into lock ranking order
    • cl/229480 runtime: added several new lock-rank partial order edges
    • cl/231463 runtime: add one extra lock ranking partial edge
    • cl/233599 runtime: add a lock partial order edge (assistQueue -> mspanSpecial)
    • cl/236137 runtime: add three new partial orders for lock ranking
  • issue/40677 runtime: lock held checking

Builder

Back To Top

Modules

Back To Top

gopls

  • design/gopls-workspace Heschi Kreinick, Rebecca Stambler. Proposal: Multi-project gopls workspaces. Mar 9, 2020.

Back To Top

Testing, x/perf

  • design/subtests Marcel van Lohuizen. testing: programmatic sub-test and sub-benchmark support. September 2, 2015.
    • issue/12166 proposal: testing: programmatic sub-test and sub-benchmark support
  • design/gotest-bench Russ Cox, Austin Clements. Proposal: Go Benchmark Data Format. February 2016.
    • issue/14313 cmd/go: decide, document standard benchmark data format
  • issue/20875 testing: consider calling ReadMemStats less during benchmarking
  • issue/27217 testing: tiny benchmark with StopTimer runs forever
  • issue/41637 testing: benchmark iteration reports incorrectly
  • issue/41641 testing: inconsistent benchmark measurements when interrupts timer
  • design/gotest-json Nodir Turakulov. Proposal: -json flag in go test. 2016-09-14.
  • design/testing-helper Caleb Spare. Proposal: testing: better support test helper functions with TB.Helper. 2016-12-27
    • issue/4899 testing: add t.Helper to make file:line results more useful
  • design/fuzzing Katie Hockman. Design Draft: First Class Fuzzing
  • issue/43744 testing: benchmark unit properties
  • issue/48803 all: Go compiler/runtime performance monitoring system
  • issue/49121 x/perf/storage: support postgres for db

Back To Top

Runtime Core

Scheduler

  • paper/work-steal Robert D. Blumofe and Charles E. Leiserson. 1999. Scheduling multithreaded computations by work stealing. J. ACM 46, 5 (September 1999), 720-748.
  • cl/sched-m-1 Russ Cox, Clean up scheduler. Aug 5, 2008.
  • cl/sched-m-n things are much better now, Nov 11, 2009.
  • design/go11sched Dmitry Vyukov. Scalable Go Scheduler Design Doc, 2012
  • design/sched-preempt-dmitry Dmitry Vyukov. Go Preemptive Scheduler Design Doc, 2013
  • design/sched-numa Dmitry Vyukov, NUMA-aware scheduler for Go. Sep 2014.
  • design/go15gomaxprocs Russ Cox. Go 1.5 GOMAXPROCS Default. May 2015.
  • doc/go17sched Ian Lance Taylor. How does the golang scheduler work? July 16, 2016.
  • design/sched-preempt-austin Austin Clements. Proposal: Non-cooperative goroutine preemption. Jan 2019.
    • cl/43050 cmd/compile: loop preemption with fault branch on amd64. May 09, 2019.
    • issue/10958 runtime: tight loops should be preemptible
    • issue/24543 runtime: non-cooperative goroutine preemption
    • issue/36365 runtime: clean up async preemption loose ends
  • issue/14592 runtime: let idle OS threads exit
  • issue/18237 runtime: scheduler is slow when goroutines are frequently woken
  • issue/20395 runtime: terminate locked OS thread if its goroutine exits
  • issue/20458 proposal: runtime: pair LockOSThread, UnlockOSThread calls
  • issue/21827 runtime: big performance penalty with runtime.LockOSThread
  • issue/27345 runtime: use parent goroutine's stack for new goroutines
  • issue/28808 runtime: scheduler work stealing slow for high GOMAXPROCS
  • issue/32113 runtime: optimization to reduce P churn.
  • issue/43997 runtime: non-spinning Ms spin uselessly when work exists
  • issue/44313 runtime: stopped Ms can't become dedicated or fractional GC workers
  • issue/51071 runtime: investigate possible Go scheduler improvements inspired by Linux Kernel's CFS

Back To Top

Execution Stack

Back To Top

Memory Management

Allocator

A quick history about the Go's memory allocator: Russ Cox first implements the memory allocator based on tcmalloc for Go 1, mcache is cached on M. Then he revised the allocator to allow user code to use 16GB memory and later allows 128GB. However, the allocator (including scavenger) was suffered from massive lock contentions and does not scale. After Dmitry's scalable runtime scheduler, the allocator can allocate directly from P (with much less) lock contentions. In the meantime, the scavenger is migrated from an independent thread into the system monitor thread. Now, Michael is actively working on improving the memory allocator's scalability, such as migrating scavenger to user threads, bitmap-based page allocator, scalable mcentral.

  • doc/tcmalloc Sanjay Ghemawat, Paul Menage. TCMalloc : Thread-Caching Malloc. Google Inc., 2009
  • design/go113scavenge Michael Knyszek. Proposal: Smarter Scavenging. 2019-05-09.
    • issue/30333 runtime: smarter scavenging
    • issue/32012 runtime: background scavenger is overzealous with small heaps
    • issue/31966 runtime: background scavenger can delay deadlock detection significantly
    • issue/34047 runtime: potential deadlock cycle caused by scavenge.lock
    • issue/34048 runtime: scavenger pacing fails to account for fragmentation
    • issue/35788 runtime: scavenger not as effective as in previous releases
    • issue/36521 runtime: performance degradation in go 1.12
    • issue/36603 runtime: sysUsed often called on non-scavenged memory
  • design/go114pagealloc Michael Knyszek, Austin Clements. Proposal: Scaling the Go page allocator. 2019-10-18.
    • issue/35112 runtime: make the page allocator scale
    • cl/200439 runtime: place lower limit on trigger ratio
  • issue/8885 runtime: consider adding 24-byte size class.
  • issue/37487 runtime: improve mcentral scalability
    • cl/221182 runtime: add new mcentral implementation
  • issue/18155 runtime: latency in sweep assists when there's no garbage
  • issue/19112 runtime: deadlock involving gcControllerState.enlistWorker
  • issue/23687 runtime: use MADV_FREE on linux as well
    • cl/135395 runtime: use MADV_FREE on Linux if available
  • issue/29707 cmd/trace: failed to parse trace: no consistent ordering of events possible
  • issue/35954 runtime: handle hitting the top of the address space in the allocator more gracefully
  • issue/37927 runtime: GC pacing exhibits strange behavior with a low GOGC
  • issue/38130 runtime: incorrect sanity checks in the page allocator
  • issue/38404 runtime: STW GC stops working on arm64/mips64le
  • issue/38605 runtime: pageAlloc.allocToCache updates pageAlloc.searchAddr in an invalid way
  • issue/38617 runtime: scavenger freezes up in Go 1.14 in Windows due to coarse time granularity
  • issue/38966 runtime: aix-ppc64 builder is failing with "bad prune", "addr range base and limit are not in the same memory segment" fatal errors
  • issue/39128 runtime: linux-mips-rtrk builder consistently failing with "bad prune" as of CL 233497
  • issue/40191 runtime: pageAlloc.searchAddr may point to unmapped memory in discontiguous heaps, violating its invariant
  • issue/40457 runtime: runqputbatch does not protect its call to globrunqputbatch
  • issue/40641 runtime: race between stack shrinking and channel send/recv leads to bad sudog values
  • issue/42330 runtime: default to MADV_DONTNEED on Linux
    • cl/267100 runtime: default to MADV_DONTNEED on Linux
  • issue/46787 runtime: provide Pinner API for object pinning
  • issue/51317 proposal: arena: new package providing memory arenas

Back To Top

Collector

  • paper/on-the-fly-gc Edsger W. Dijkstra, Leslie Lamport, A. J. Martin, C. S. Scholten, and E. F. M. Steffens. 1978. On-the-fly garbage collection: An exercise in cooperation. Commun. ACM 21, 11 (November 1978), 966–975.
  • paper/yuasa-barrier T. Yuasa. 1990. Real-time garbage collection on general-purpose machines. J. Syst. Softw. 11, 3 (March 1990), 181-198.
  • design/go13gc Dmitry Vyukov. Simpler and faster GC for Go. July 16, 2014
  • design/go14gc Richard L. Hudson. Go 1.4+ Garbage Collection (GC) Plan and Roadmap. August 6, 2014.
  • design/go15gcpacing Austin Clements. Go 1.5 concurrent garbage collector pacing. Mar 10, 2015.
  • discuss/gcpacing Austin Clements et al. Discussion of "Proposal: Garbage collector pacing". March 10, 2015.
  • issue/11970 runtime: replace GC coordinator with state machine
  • design/sweep-free-alloc Austin Clements. Proposal: Dense mark bits and sweep-free allocation. Sep 30, 2015.
  • issue/12800 runtime: replace free list with direct bitmap allocation
  • design/decentralized-gc Austin Clements. Proposal: Decentralized GC coordination. October 25, 2015.
  • issue/12967 runtime: shrinkstack during mark termination significantly increases GC STW time
  • issue/14951 runtime: mutator assists are over-aggressive, especially at high GOGC
  • design/eliminate-rescan Austin Clements, Rick Hudson. Eliminate STW stack re-scanning. October 21, 2016.
  • design/concurrent-rescan Austin Clements, Rick Hudson. Proposal: Concurrent stack re-scanning. Oct 18, 2016.
    • issue/17505 runtime: perform concurrent stack re-scanning
  • design/soft-heap-limit Austin Clements. Proposal: Separate soft and hard heap size goal. October 21, 2017.
  • issue/22460 runtime: optimize write barrier
  • design/roc Request Oriented Collector (ROC) Algorithm
  • doc/ismm-gc Rick Hudson. Getting to Go: The Journey of Go's Garbage Collector. 12 July 2018.
  • discuss/ismm-gc Garbage Collection Slides and Transcript now available
  • design/simplify-mark-termination Austin Clements. Proposal: Simplify mark termination and eliminate mark 2. Aug 9, 2018.
    • issue/26903 runtime: simplify mark termination and eliminate mark 2
  • design/gcscan Proposal: GC scanning of stacks
    • issue/22350 cmd/compile: compiler can unexpectedly preserve memory,
  • issue/27993 runtime: error message: P has cached GC work at end of mark termination
  • issue/37116 runtime: 10ms-26ms latency from GC in go1.14rc1, possibly due to 'GC (idle)' work
  • issue/42430 runtime: GC pacer problems meta-issue
    • issue/39983 runtime: idle GC interferes with auto-scaling
    • issue/17969 runtime: aggressive GC completion is disruptive to co-tenants
    • issue/14812 runtime: GC causes latency spikes
    • issue/40460 runtime: goroutines may allocate beyond the hard heap goal
    • issue/29696 proposal: runtime: add way for applications to respond to GC backpressure
    • issue/23044 proposal: runtime: add a mechanism for specifying a minimum target heap size
  • issue/42642 runtime: multi-ms sweep termination pauses (second edition)
  • issue/44163 runtime: remove idle GC workers
  • issue/44167 runtime: GC pacer redesign
  • issue/44309 proposal: runtime/debug: user-configurable memory target
  • issue/48409 proposal: runtime/debug: soft memory limit
  • issue/45894 runtime: mark termination is slow to restart mutator
  • issue/45315 runtime: runtime.GC can return without finishing sweep
  • issue/49075 runtime: possible memory corruption
  • issue/52433 runtime: heap goal overrun due to scheduler delays in mark termination

Back To Top

Statistics

Back To Top

Memory model

The Go memory model consists the following parts:

  • Memory order regarding atomic operations

  • Memory order regarding the sync package APIs

  • Memory order regarding runtime mechanism (i.e. Object finalizer)

  • doc/refmem Rob Pike and Russ Cox. The Go Memory Model. February 21, 2009.

  • issue/4947 cmd/cc: atomic intrinsics

  • issue/5045 doc: define how sync/atomic interacts with memory model

  • issue/7948 doc: define how sync interacts with memory model

  • issue/9442 doc: define how finalizers interact with memory model

  • issue/33815 doc/go_mem: "hello, world" will not always be printed twice

  • cl/75130045 doc: allow buffered channel as semaphore without initialization

  • doc/gomem Russ Cox. Go’s Memory Model. February 25, 2016.

  • doc/go2017russ Russ Cox. My Go Resolutions for 2017: Memory model. January 18, 2017.

  • doc/atomic-bug Package atomic

  • discuss/atomic-mem-order specify the memory order guarantee provided by atomic Load/Store

  • issue/37355 runtime/race: running with -race misses races (mismatch with memory model)

    • cl/220419 runtime: swap the order of raceacquire() and racerelease
    • issue/42598 runtime: apparent false-positive race report for a buffered channel after CL 220419
    • cl/271987 runtime: check channel's elemsize before calling race detector
    • [paper/fava2020fix] Fava, Daniel Schnetzer. "Finding and Fixing a Mismatch Between the Go Memory Model and Data-Race Detector." International Conference on Software Engineering and Formal Methods. Springer, Cham, 2020.

  • doc/mm Russ Cox. Memory Models. June, 2021.

    • doc/hwmm Russ Cox. Hardware Memory Models. June 29, 2021.
    • doc/plmm Russ Cox. Programming Language Memory Models. July 6, 2021.
    • doc/gomm Russ Cox. Updating the Go Memory Model. July 12, 2021.
    • discuss/47141 Updating the Go memory model.
    • issue/50860 proposal: sync/atomic: add typed atomic values
    • issue/50859 proposal: doc: update Go memory model
    • doc/out-of-thin-air Hans-J. Boehm. P1217R2: Out-of-thin-air, revisited, again. 2019-06-16.
    • doc/cpp-memory-model Hans-J. Boehm, Sarita V. Adve. Foundations of the C++ Concurrency Memory Model. May 21, 2008.
    • cl/381315 doc: update Go memory model
    • cl/381316 runtime, sync, sync/atomic: document happens-before guarantees

Back To Top

ABI

  • design/cgo-pointers Ian Lance Taylor. Proposal: Rules for passing pointers between Go and C. October, 2015
    • issue/12416 cmd/cgo: specify rules for passing pointers between Go and C
  • design/internal-abi Austin Clements. Proposal: Create an undefined internal calling convention. Jan 14, 2019.
    • issue/27539 proposal: runtime: make the ABI undefined as a step toward changing it.
    • issue/31193 cmd/compile, runtime: reduce function prologue overhead
  • design/register-call Austin Clements, with input from Cherry Zhang, Michael Knyszek, Martin Möhrmann, Michael Pratt, David Chase, Keith Randall, Dan Scales, and Ian Lance Taylor. Proposal: Register-based Go calling convention. Aug 10, 2020.
  • issue/18597 proposal: cmd/compile: define register-based calling convention
  • issue/40724 proposal: switch to a register-based calling convention for Go functions
    • cl/266638 reflect,runtime: use internal ABI for selected ASM routines, attempt 2
    • cl/259445 cmd/compile,cmd/link: initial support for ABI wrappers
  • design/internal-abi-spec Go internal ABI specification.

Back To Top

Misc

  • issue/20135 runtime: shrink map as elements are deleted
  • issue/48687 runtime: enhance map cacheline efficiency

Standard Library

syscall

Back To Top

os, io, io/fs, embed

In Go 1.16, tons of major rework and improvements surround the new os/fs package.

  • design/draft-iofs Russ Cox, Rob Pike. File System Interfaces for Go — Draft Design. July 2020.
    • issue/13473 proposal: os: Stdin, Stdout and Stderr should be interfaces
    • issue/14106 proposal: os: File should be an interface
    • issue/19660 proposal: io/ioutil: rename to io/fileio or similar
    • issue/40025 proposal: io/ioutil: move Discard, NopCloser, ReadAll to io
    • issue/42027 proposal: path/filepath: add WalkDir (Walk using DirEntry)
    • issue/41190 io/fs: add file system interfaces
    • issue/41467 os: add ReadDir method for lightweight directory reading
    • issue/41974 proposal: io/fs, filepath: add more efficient Walk alternative
    • issue/42026 proposal: os: add ReadDir, ReadFile, WriteFile, CreateTemp, MkdirTemp & deprecate io/ioutil
    • issue/43223 proposal: io/fs, net/http: define interface for automatic ETag serving
  • design/go116embed Embedded files - Russ & Braid
    • issue/41191 embed, cmd/go: add support for embedded files
    • issue/42321 embed: warn about dotfiles in embed.FS documentation
    • issue/42328 proposal: cmd/go: avoid surprising inclusion of "hidden" files when using //go:embed
    • issue/43216 proposal: embed: remove support for embedding directives on local variables
    • issue/43217 proposal: embed: define String and Bytes type aliases that must be used with //go:embed
    • issue/43218 embed: resolve string, []byte issues
    • issue/44166 io/fs,os: fs.ReadDir with an os.DirFS can produce invalid paths
    • issue/42322 io/fs: add func Sub(fsys FS, dir string) FS
  • issue/45757 proposal: io/fs: add writable interfaces

Back To Top

go/*

  • doc/gotypes Alan Donovan. go/types: The Go Type Checker.
  • talk/gotypes Alan Donovan. Using go/types for Code Comprehension and Refactoring Tools. October 2, 2015.
  • design/modular-interface Alan Donovan. Proposal: a common interface for modular static analyses for Go. 9 Sep 2018
    • cl/134935 go/analysis: a new API for analysis tools

Back To Top

sync

  • design/percpu-sharded Sanjay Menakuru. Proposal: percpu.Sharded, an API for reducing cache contention. Jun 12, 2018.
    • issue/18802 proposal: sync: support for sharded values
  • issue/37142 sync: shrink types in sync package

Back To Top

Map

  • issue/21031 sync: reduce pointer overhead in Map
  • issue/21032 sync: reduce (*Map).Load penalty for Stores with new keys
  • issue/21035 sync: reduce contention between Map operations with new-but-disjoint keys
  • issue/37033 runtime: provide centralized facility for managing (c)go pointer handles
  • issue/51972 sync: add new Map methods CompareAndSwap, CompareAndDelete, Swap

Back To Top

Pool

  • discuss/add-pool gc-aware pool draining policy
  • issue/4720 sync: add Pool type. Jan 28, 2013.
  • cl/46010043 sync: scalable Pool. Jan 24, 2014.
  • cl/86020043 sync: less agressive local caching in Pool. Apr 14, 2014.
  • cl/162980043 sync: release Pool memory during second and later GCs. Oct 22, 2014.
  • issue/8979 sync: Pool does not release memory on GC
  • issue/22331 runtime: clearpools causes excessive STW1 time
  • issue/22950 sync: avoid clearing the full Pool on every GC.
    • cl/166960 sync: use lock-free structure for Pool stealing.
    • cl/166961 166961: sync: smooth out Pool behavior over GC with a victim cache.
  • issue/24479 sync: eliminate global Mutex in Pool operations

Back To Top

Mutex, RWMutex

  • cl/4631059 runtime: replace Semacquire/Semrelease implementation.
  • issue/9201 proposal: sync: prohibit unlocking mutex in a different goroutine
  • issue/13086 runtime: fall back to fair locks after repeated sleep-acquire failures.
  • issue/17973 sync: RWMutex scales poorly with CPU count
    • cl/215361 sync: Implement a version of RWMutex that can avoid cache contention

Back To Top

Groups

  • cl/134395 errgroup: rethink concurrency patterns
    • cl/131815 errgroup: handle runtime.Goexit from child goroutines
    • issue/15758 testing: complain loudly during concurrent use of T.FatalX and T.SkipX
    • issue/25448 proposal: promote panic(nil) to non-nil panic value

atomic

  • issue/8739 runtime,sync/atomic: unify API for runtime/internal/atomic and sync/atomic
  • issue/20164 proposal: atomic: add (*Value).Swap
  • discuss/atomic-value
  • issue/36606 proposal: cmd/compile: make 64-bit fields be 64-bit aligned on 32-bit systems, add //go:packed directive on structs
  • issue/37262 runtime: can't atomic access of first word of tiny-allocated struct on 32-bit architecture

Back To Top

time

  • design/monotonic-time Russ Cox. Proposal: Monotonic Elapsed Time Measurements in Go. January 26, 2017.
    • issue/12914 time: use monotonic clock to measure elapsed time
  • Scalable Timer
    • cl/34784 runtime: improve timers scalability on multi-CPU systems
    • issue/6239 runtime: make timers faster.
    • issue/15133 runtime: timer doesn't scale on multi-CPU systems with a lot of timers
    • issue/27707 time: excessive CPU usage when using Ticker and Sleep.
  • Followup latency issues
    • issue/18023 runtime: unexpectedly large slowdown with runtime.LockOSThread
    • issue/25471 time: Sleep requires ~7 syscalls
    • issue/38070 runtime: timer self-deadlock due to preemption point
    • issue/36298 net: 1.14 performance regression on mac
    • issue/38860 runtime: CPU bound goroutines cause unnecessary timer latency
      • cl/216198 runtime: add goroutines returned by poller to local run queue
      • cl/232199 runtime: steal timers from running P's
      • cl/232298 runtime: reduce timer latency
    • issue/44343 runtime: time.Sleep takes more time than expected
    • issue/44868 time, runtime: zero duration timer takes 2 minutes to fire

Back To Top

context

  • issue/8082 proposal: spec: represent interfaces by their definition and not by package and name
  • issue/14660 proposal: context: new package for standard library
  • issue/16209 Proposal: relaxed rules for assignability with differently-named but identical interfaces
  • issue/20280 proposal: io: add Context parameter to Reader, etc.
  • issue/21355 proposal: Replace Context with goroutine-local storage
  • issue/24050 testing: test with child process sometimes hangs on 1.10; -timeout not respected
  • issue/27982 proposal: Go 2: bake a cooperative goroutine cancellation mechanism into the language
  • issue/28342 proposal: Go 2: update context package for Go 2
  • issue/29011 proposal: Go 2: use structured concurrency
  • doc/context-go-away Michal Štrba. Context should go away for Go 2. 2017/08/06
  • doc/context Go Concurrency Patterns: Context.
  • doc/context-isnt-for-cancellation Dave Cheney. Context isn’t for cancellation. August 20, 2017.
  • issue/42564 context: cancelCtx exclusive lock causes extreme contention
  • issue/51365 proposal: context: add APIs for writing and reading cancelation cause

Back To Top

encoding

  • design/go12encoding Russ Cox. Go 1.2 encoding.TextMarshaler and TextUnmarshaler. July 2013.
  • design/go12xml Russ Cox. Go 1.2 xml.Marshaler and Unmarshaler. July 2013.
  • design/natural-xml Sam Whited. Proposal: Natural XML. 2016-09-27
    • issue/13504 encoding/xml: add generic representation of XML data
  • design/zip Russ Cox. Proposal: Zip-based Go package archives. February 2016
    • issue/14386 proposal: use zip format inside .a and .o files
  • design/xmlstream Sam Whited. Proposal: XML Stream. Mar 9, 2017.
    • issue/19480 encoding/xml: add decode from TokenReader, to enable stream transformers
  • design/raw-xml Sam Whited. Proposal: Raw XML Token. Sep 1, 2018.

Back To Top

image, x/image

The following issues are surrounding by the color management of the image standard library. At the moment, Go's image library doesn't read or write meta information from an image in the encoding or decoding phase. Therefore the color information could go wrong while processing an image such as scaling in a non-linear sRGB space. A universal solution is to design image metadata APIs to aware the color profile in an encoded image file.

  • issue/11420 x/image/draw: color space-correct interpolation
    • issue/20613 image/png: don't ignore PNG gAMA chunk
    • issue/27830 proposal: image: decoding options
    • cl/253497 x/image/draw: gamma corrected non linear interpolation
    • issue/37188 image/color: documentation doesn't include links to relevant color theory resources
  • issue/33457 proposal: image: add generic metadata support for jpeg, gif, png
  • issue/44808 image, image/draw: add interfaces for using RGBA64 directly
  • issue/46395 image/draw: increase performances by applying special case if mask is *image.Alpha
  • issue/8055 image: decode / resize into an existing buffer
  • issue/11793 image/color: NRGBA(64).RGBA() optimization
  • issue/15759 image: optimize Image.At().RGBA()
  • issue/20851 image: Decode drops interfaces
  • issue/24499 image/jpeg: Decode is slow

Back To Top

math

  • issue/8037 cmd/internal/obj/x86: add fma
  • issue/17895 spec: allow the use of fused multiply-add floating point instructions
    • cl/40391 spec: clarify use of fused-floating point operations
  • issue/24813 math/bits: add extended precision Add, Sub, Mul, Div
  • issue/25819 math: add guaranteed FMA

Back To Top

Mobile

Back To Top

misc

  • design/mobile-audio Jaana Burcu Dogan. Proposal: Audio for Mobile. November 30, 2015.
  • design/localization Marcel van Lohuizen. Proposal: Localization support in Go. Jan 28, 2016.
  • design/unbounded-queue Christian Petrin. Proposal: Built in support for high performance unbounded queue. October 2, 2018
  • design/lockfile Adrien Delorme. Proposal: make the internal lockedfile package public. 2019-10-15.
  • design/cidr Rudi Kramer, James Forrest. Proposal: Add support for CIDR notation in no_proxy variable. 2017-07-10
    • issue/16704 net/http: considering supporting CIDR notation in no_proxy env variable
  • design/dns Sam Whited. Proposal: DNS Based Vanity Imports. 2018-06-30.
    • issue/26160 proposal: use DNS TXT records for vanity import paths
  • issue/46518 net/netip: add new IP address package, use in net
  • issue/53171 proposal: add package for using SIMD instructions

Back To Top

Unclassified But Relevant Links

Back To Top

Fun Facts

  • cl/1 Brian Kernighan. Go's first commit. Jul 19, 1972.
  • issue/9 I have already used the name for MY programming language. Nov 11, 2009
  • issue/2870 9 is prime if it's a hot day. Feb 3, 2012
  • doc/gophercount How Many Go Developers Are There?. November 1, 2019.
  • discuss/google-owns-go Russ Cox's response on "Go is Google's language, not ours"

Back To Top

Acknowledgements

The document author would like to first thank the TalkGo community creator Mai Yang's champion sponsorship for the golang.design initiative. His creation of the TalkGo significantly changed the Go community in China. He is also a great person that is actively contributing to all kinds of Go related projects.

It is also important to thank the continuing, inspiring discussion and sharing with the TalkGo community core members qcrao, aofei, and eddycjy.

The document would not be organized without all of the supports from them.

Back To Top

License

golang.design/history | CC-BY-NC-ND 4.0 © changkun

← Metadata

1.0k
Stars
47
Forks
Watchers

Owner

verified publisher icon golang-design

Metadata

📝 Go: A Documentary | https://golang.design/history

Back