pkg directory should not be recommended for use

[theckman]

The general consensus of the Go community has been converging on the idea that the pkg/ directory is a useless abstraction / indirection. Instead, people are encouraged to either put the packages in the top-level of the repository, or to create a repository structure that structures things based on their primary functionality of domain in the business logic.

So while in your example you may encourage people to have the following layout:

pkg/
  server/
    handlers/
  poller/
    config/
    manager/

We instead would prefer:

server/
  handlers/
poller/
  config/
  manager/

This may seem innocuous, but because this project has asserted itself as some sort of Go standards body what is being recommended here is being taken seriously. In the Slack workspace, where there are over 30k registered users, we quite often need to correct people who are using pkg/ and have them ignore the recommendations here. Some of the things recommended here are good, and I'd like to be able to use them as a resource. That said, considering the current pkg/ recommendation we're hard-pressed to use this as a resource because we have to tell people to ignore certain parts.

[rburmorrison]

@theckman Interesting! I'll certainly consider this adopting this. What is the general reason that using the pkg/ directory is frowned upon, may I ask? Is it due to long import paths? I tend to like it because I can name my root folder what I want as to not conflict with my other github projects, and keep the actual package name separate within the pkg/ directory. I also like that I can keep configuration files and READMEs separate from my source code. But it is certainly a long import path. Also, do you tend to agree with the cmd/ directory, as it is similar to the usage of the pkg/ directory? Thanks!

[kcq]

Can you be a bit more specific about the 'we' part. Who exactly are you talking about. Is there an official community consensus around this (controversial) pattern? Is there a consensus from the official go dev team? At least, the last interaction around it on the gophers slack channel is limited mostly to you saying don't use the 'pkg' pattern :-) There might be other threads and discussions I didn't see and that's why I'm asking.

This repo attempts to capture the existing or emerging patterns in the Go ecosystem / project community. The 'pkg' pattern is used in a number of high profile and popular projects. Some of those projects might not be great examples others should follow, but it would be a bit suspicious to say that all projects leveraging the 'pkg' pattern are bad examples :-) Of course, it's not something that the majority of projects out there do and it appears there are some strong feelings against it, but i'm not sure if there's a unanimous agreement that it's a bad pattern. I definitely agree that it's a confusing pattern for new go devs.

I'll be happy to add more context and the necessary references to the emerging community recommendations.

[theckman]

@rburmorrison I'll start by saying I'm a proponent of cmd/, especially when you have multiple binaries being built. It gives you a good starting point to understand how the library is imported / consumed. 👍

For pkg/, there's a few reasons why I think it's started to fall out of favor. The first being that any packages not encapsulated within an internal directory are considered publicly consumable. You can import them, and the compiler will allow you to build the software. And so whether or not it's included in pkg people can consume it. If you don't want to expose it, you should absolutely use the internal package convention to do so. This will cause the compiler to fail to build because it's an internal package.

The second reason kind of has to do with long paths, but it's also that pkg doesn't provide a useful grouping for me. If I want to see where the source code lives, I've tools that can easily walk the tree and tell me that. Even then I generally don't use them, because I'm either already familiar with the library or I'm reading code that uses the library and deducing where the code is based on import statements.

[theckman]

@kcq I'm speaking in terms of what has become a general consensus in the Go Slack and Forums areas. The Go authors themselves have been absent in providing guidance on this topic, rightfully so.

There are discussions beyond ones involving me (based on the interactions I've seen), and my opinion was shifted from liking pkg/ to not after having become involved in the Go community itself. I think I shifted to this opinion well-before the date of the first commit of this repo.

Library code that's safe to use by external applications (e.g., /pkg/mypubliclib).

I think one issue I have with explanations like this, is that the internal/ directory is where code that's unsafe for use by external applications is meant to go. And so if it's outside of an internal directory, the author is saying it's safe to use as an external application no matter the top-level directory. It may be better to say it's one location where people sometimes put library code, and make no mention of the external consumption part?

[nineinchnick]

I don't know if a Tweet can/should be used as source but Brad Fitzpatrick states pkg is legacy: https://twitter.com/bradfitz/status/1039512487538970624 There is definitely some momentum though.

[kcq]

@nineinchnick yes, it's a 'legacy' pattern for Go itself. Brad's tweet provides a nice background about the origins of the pattern :)

[hron84]

@theckman While I'm not a Go developer, I do know a little bit about development in general terms. You said:

think one issue I have with explanations like this, is that the internal/ directory is where code that's unsafe for use by external applications is meant to go. And so if it's outside of an internal directory, the author is saying it's safe to use as an external application no matter the top-level directory.

The problem is you made a direct conclusion that if something is not in internal directory then it's safe to use by external application/project which is not neccessarily true. Think about cmd directory, while the proposal says only importing things and launching things should go here, things could be more difficult and this code is still not meant to public use/re-use.

I personally like the idea of pkg directory, even if I come from Java/Ruby world, because if something is in this directory it clearly statest that code is intended for public re-use and if something is located at different, it does not. Note, I emphasized word public because if some projects are share code partially internally this not directly means that code is also meant/intended to public reuse. So, there could be a gray zone and pkg directory makes things more clear. If something is in this directory, use it freely. If not, expect some consequences (not neccessarily legal).

[theckman]

@hron84 In Go you can put it at the top level of a package so all things inside of that repository can use it. If you are looking for code reuse across repos, that means you expect it to not be internal and consumed externally. If you consider that, I don't think my statement is a stretch.

[rakyll]

pkg is not a canonical practice to organize "importable" packages. One side effect of representing it here is that I hear a lot from others that I should reorg my packages as suggested as here.

it clearly statest that code is intended for public re-use

I don't think there is a widely accepted rule for this. This is the first time I am hearing this intention and I personally know very few people who thinks "pkg" makes a package intended for public use. Any importable package is for public use. Anything else goes into internal. Note that internal package is not just a convention, it is supported by the Go tool.

[egorse]

/cc @peterbourgon @thockin

[peterbourgon]

The pkg subdirectory is a way to sequester Go source files into well-known locations in a repository. That is, all Go files are guaranteed to live in ./{cmd,pkg,internal}/*, with cmd being for all package main, pkg for reusable packages, and internal for non-reusable packages. This property is useful on its own, but it is especially useful in repositories that have lots of other types of files, for example build and hack scripts, config files, static assets, or source code written in other languages.

This value proposition seems clear and unambiguous to me. I don't understand why some Go programmers are so vocal about squashing it. I don't think it is mandatory, and I wouldn't suggest advertising it that way, but it's clearly useful and I would push back against (or at least ignore) anyone who suggested it not be used.

[theckman]

@peterbourgon It's not clear and unambiguous to me, because it's honestly not clear what value it adds beyond cognitive overhead and unnecessary abstractions. To me it's just noise. I don't really care what directories my Go source code is in. What makes you care?

My editor has jump to definition, and I have other tooling I can use to discover files based on their declared types. I also have an easy way to open any file in my current project using a simple keyboard shortcut. It's been superseded by better tooling, functionality, editor integrations, and assumptions by the Go toolkit, and we should retire it as a relic of times past

I also have issue with this quote in the README:

pkg/: Library code that's ok to use by external applications

No, anything that's NOT in internal is ok to use by external applications.

[peterbourgon]

I don't really care what directories my Go source code is in. What makes you care?

If I'm looking for Go code, I know where to look. Equally importantly, I know where not to look: is hack/ a poorly-named package, or a collection of scripts? With pkg I know for sure. Even more concretely, when I run commands that target Go source files or source trees, I can point them in the right place, without paying the costs of tree walking or stat calling on my (potentially very large) tree.

For example: go test ./{cmd,pkg,internal}/... versus go test ./... can be a huge difference in wall-clock time on repos with lots of non-Go-source files.

[pkg has] been superseded by better tooling, functionality, editor integrations, and assumptions by the Go toolkit, and we should retire it as a relic of times past.

You are affirming the consequent.

[theckman]

Hmm, that makes me less convinced. If I'm looking for Go code I generally have an entry point. A function, a file, some string to search for, etc. I use my editor's tools to do the work beyond that. It seems like a pretty inefficient workflow to just go spelunking in repos looking for Go files without any other way to drill-down to what you want to find.

If someone wants to keep all the go code together, I'm :+1: on a top-level go directory. I'm :-1: on a list of arbitrary locations which may or may not have Go code, and which may or may not also indicate the public nature of those packages.

[peterbourgon]

Well, I guess it's moot, really: despite the rather lofty implications of the name of this GitHub organization and repo, there is no standards body which canonizes an e.g. repo layout, only the collective opinions of Go programmers. I'm happy to let time be the judge of this practice. I'll bow out now.

[rakyll]

(I don't have any strong opinions whether this can be a canonical practice and please say so if this is a bikeshedding discussion.)

I think the question is whether location and organization should tell anything about a Go package or not. Organizing packages under pkg/ makes their import path include "pkg" and advertise that they are not main packages. It might be an arbitrary distinction which has a lot of impact on the package identity. Tools might benefit from the distinction but there are no tools that blesses this convention (yet).

[giefferre]

There's no mention of cmd, pkg or internal in the official Go documentation.

What I can understand from the doc is: create packages only if you really need to. That basically means that you should avoid exporting internal structs and methods, possibly keeping them together with the main package.

[kcq]

@giefferre the internal directory is actually in the official docs :-) It's referenced in the Go 1.4 release notes: https://golang.org/doc/go1.4#internalpackages

[kcq]

@peterbourgon It's ok if not everybody finds the pkg pattern useful. A lot more people do though. Thank you for sharing your arguments for it! There have been many precedents of similar patterns created in many different languages... The hungarian notation is one of them (not saying it's good or bad :-)) where you have an unofficial construct that indicates the intent.

[jamelt]

A very new Go developer, but as an experienced developer. Here are my thoughts. I generally like the idea of pkg/ but I'd say doing things the community-accepted way is more important. I wish there was a definitive answer.

For pkg/ Folder:

Pros

• 👍 Less files in a top level directory, less friction to understanding a foreign codebase. Standards help newcomers navigate.
• 👍 Fewer homes for source files is generally good, if only proven empirically. We see this in many other languages. Personally, I have a strong distaste for source code being mixed in with .sh, .json, .md files, etc.
• 👍 Accepts the reality that projects/repos house more than just source code, things such as: default configuration, install scripts, etc.
• 👍 Consistency of code organization across a broader community than yourself or organization.

Cons

• 👎 Not official guidance, so there's always a chance for this to be up for debate. Detracts from time otherwise spent productively.
• 👎 Keeping package names 100% meaningful, adding "pkg/" to all of them is repetitive and noisy. Although this would go away, if Go officially supported this. If I was writing an open-source library, the pkg layout would probably be a non-starter for me.
• 👎 Moves away from the elegance of packages in Go and their naming seamlessly merging into your folder/repo hierarchy.
• 👎 Objectively unnecessary: the only benefit is cosmetic.

[gamorejon]

A very new Go developer, but as an experienced developer. Here are my thoughts. I generally like the idea of pkg/ but I'd say doing things the community-accepted way is more important. I wish there was a definitive answer.

For pkg/ Folder:

Pros

• 👍 Less files in a top level directory, less friction to understanding a foreign codebase. Standards help newcomers navigate.
• 👍 Fewer homes for source files is generally good, if only proven empirically. We see this in many other languages. Personally, I have a strong distaste for source code being mixed in with .sh, .json, .md files, etc.
• 👍 Accepts the reality that projects/repos house more than just source code, things such as: default configuration, install scripts, etc.
• 👍 Consistency of code organization across a broader community than yourself or organization.

Cons

• 👎 Not official guidance, so there's always a chance for this to be up for debate. Detracts from time otherwise spent productively.
• 👎 Keeping package names 100% meaningful, adding "pkg/" to all of them is repetitive and noisy. Although this would go away, if Go officially supported this. If I was writing an open-source library, the pkg layout would probably be a non-starter for me.
• 👎 Moves away from the elegance of packages in Go and their naming seamlessly merging into your folder/repo hierarchy.
• 👎 Objectively unnecessary: the only benefit is cosmetic.

So first I like to gently push back on a few of these points:

• "Keeping package names 100% meaningful": There is no pkg package. If there is a directory with no go files in it declaring a package there isn't one. When you import "github.com/xxx/yyy" there is no "github.com" package, it's just part of the import path. One of the commenters above mentioned something about unnecessary abstraction, but there is no actual abstraction added, just a "different" import path which indicates the direction of dependencies in the project to developers without abstraction.
• "Moves away from the elegance of packages in Go and their naming seamlessly merging into your folder/repo hierarchy.": Go has always, intentionally used package-less import paths like "github.com" as mechanism for organization without special meaning.
• "Not official guidance, so there's always a chance for this to be up for debate.": this applies to both sides. Not using it is up for debate... in this thread.
• "Objectively unnecessary: the only benefit is cosmetic.": It's hard to know what the criterion is for unnecessary, but I suspect not coding everything in assembly language could be called purely cosmetic and objectively unnecessary in the implicit definition.

Maybe it's just the projects I've worked on, but I do think something being missed is that packages located in "pkg" also imply they have no dependencies on any packages in the rest of the project not located in pkg and it lets other developers working on a repo know not to add them. Primary, top-level library functionality the project wants to expose is still done so outside pkg.

I typically use this to incrementally pull out common patterns into their own frameworks before moving them into entirely separate repo.

So, you could call discoverability, glanceability, and organization "cosmetic" but so are a lot of language structures.

[peterbourgon]

Objectively unnecessary: the only benefit is cosmetic.

I have described in some detail the concrete and technical benefits pkg/ provides. If those benefits aren't useful to you, that's fine. But they are not purely cosmetic.

[lenkite]

I believe this repository should be renamed to non-standard-project-layout or highly-opinionated-project-layout as it no longer represents the standard Go project layout and only confuses people.

[amnonbc]

I prefer import "github.com/gorilla/mux"

to import "github.com/gorilla/pkg/mux"

The latter is less concise. The word pkg adds nothing. Of course it is a package, otherwise I would not be importing it.

Having the word pkg in import paths is convoluted, verbose and un-go-like.

[0xdeface]

if i write some code which will be used in many micro-services(for example generated protobuf + some methods) i can use pkg folder and init git submodule and develop that part near service and uc?

[Southclaws]

I've always found "pkg" useless. 1. it's a 3 letter shortening, which isn't friendly to non-English speakers who may think it's an acronym for something (P.K.G.?) 2. it's a useless name that tells you nothing about the content. Why "package"? Every directory that contains .go files is a package. It's basically the same as creating a folder called "files".

I recently changed some codebases to use pkg (to be fair we were using "src" which is equally as bad) but I feel like just moving our three subdirectories to the top level would make the most sense.

[svengreb]

This whole discussion is pointless as everyone should use the layout they like, absolutely no one should dictate how your structure looks like as long as this is not enforced by the programming language. Most of the contributors in this issue try to convince everyone to follow their preferred way which is totally selfish and the root of a lot of problems. The initial post says that pkg is a “useless abstraction / indirection“ which again is unnecessarily judgmental and not factual and also states that “people are encouraged“ to not use pkg while there is no source about this statement at all that says that people should do that.

After all everyone can use pkg whenever they want and see the need to achieve clean repository. Some say that these three characters are “too much“ and if I start to count the characters in this and other issues about this topic and divide it by 3, the result will be ridiculous about how many times you could simply write pkg instead of throwing invalid arguments into the void. In addition to that your IDE should auto-import and “write“ the massive amount of these 3 characters for you, otherwise you have a completely different problem regarding your local development setup instead of this non-sense topic.

I'm a pkg user and will never change as long as Go does not allow to define the “entrypoint“ of a package that differs from the file structure on disk which means that the file structure = package import paths logic must change.

Southclaws: It's basically the same as creating a folder called "files".

This means your project should never contain any directory because your documentations, deployment manifests, scripts and whatever is part of your repository are also only “files“.

Anyone must understand that a repository of a Go package is, just like any other project written in a different programming language, more that just your Go “files“, documentations and everything else is part of it and should be separated accordingly. Many large Go project suffer from contributors because some maintainers try to follow this non-sense “is does not feel like Go“ way of structuring which discourages potential contributors because they'll need to spend days to get an overview of the mess of data that lives in the project root. There are directories named doc which might contain documentations, but maybe also Go source code because the application does something with text parsing. They try to learn about how the project is build and looking for the build setup, but can't find it because the configurations are stored next to Go files that are also related to the configuration of the application itself. It is a total mess when there are also 150+ Go files in the repository root to know which are relevant to the application and which ones are to the project setup (e.g. linter configurations, scripts etc.). Some of the larger projects like Kubernetes also finally fixed the mess problem and make use of pkg now to prevent the project from drifting into the compulsive hoarding syndrome.

The pkg directory has one simple and valid use case: Make it clear what is Go code and what not. Everyone who don‘t want to use it can could simply move the go.mod file into a sub-directory to indicate where the actual Go code lives.

This comment is not meant to be rude and also not to criticize others but more an eye opener to finally see the whole and not only 3 characters in an important path. A project repository is more that just Go packages and no matter how the maintainers decide to structure it in the end, it is their decision and no one has the right to judge it.

[Southclaws]

This means your project should never contain any directory because your documentations, deployment manifests, scripts and whatever is part of your repository are also only “files“.

I think you missed my point haha, maybe it was poorly written.

My point was that since every directory of Go files is considered a "package" by the compiler, it doesn't tell the reader much by grouping together some packages and using a name that simply indicates that you've grouped together some packages. You can tell that by observing that there are Go files in there!

The analogy regarding "files" is more about superfluous nomenclature, nothing more. Essentially, tell your future readers what the intention is behind your code, clear and simple.

But also, I do agree that it doesn't really matter in the grand scheme of things. If you're building a product at a company, optimise for things that matter and longtermism. You don't onboard a new developer every day of the year all year round but your team works with your code for many years, so optimise for that. Which means don't waste time figuring out the "best" folder structure - your engineers will learn how to navigate regardless.

[kke]

My opinion is that pkg/ is fine in a project that's primarily something that works on its own, let's say a web browser written in go and they publish reusable packages under pkg/.

It would be great if in those cases we could think that the stuff under such project's pkg/ is something the authors specifically expect others to import from their project, something that they try hard not to change in a way that breaks compatibility.

If the project is more like a library, something that doesn't do anything on its own, like a logger for example, the whole reason for its existence is to be imported from outside. In that case, the pkg adds nothing but noise.

I want to import example.com/foofoo/superlogger, not example.com/foofoo/superlogger/pkg/superlogger. And this is how it is almost 100% of the time in the wild.

When I need to import the XML formatter for superlogger, I expect to import it from example.com/foofoo/superlogger/xml and refer to it as xml.Formatter{} and it's completely ok by me if it's in example.com/foofoo/superlogger/format/xml, but I see no reason to expect or prefer it to be in pkg/format/xml.

Maybe the purpose of this repository is to be an example for an application project layout, not a library layout.