[Feature request]: Restructure files structure

[eugeis]

What do you need?

Our root folder has too many files, and we’d like to restructure it as follows:

.
├── .github        # GitHub CI workflows
├── go/            # All Go source code
├── patterns
├── integ/         # Integration code, Docker, Nix, scripts, etc.
└── README.md

[jaredmontoya]

@eugeis I agree that compared to other projects fabric's root directory is messy and requires tidying, but having files in the root of the project isn't inherently bad. Only allowing 3 directories and a README file + a hidden .github directory is too much sacrifice for little gain.

Let's redesign project layout not based on blindly minimizing the amount of things in the project root at all costs, but on how it will be the most efficient in attracting code contributions by leveraging previous knowledge of developers and making fabric's codebase easy to understand at first glance.

Here is how our project root looks now:

Hidden files and directories

• .devcontainer
• .git
• .github
• .gitignore
• .envrc
• .dockerignore

Go code directories

• cli
• common
• core
• plugins
• restapi

Go code

• main.go
• version.go

Go dependency files

• go.mod
• go.sum
• gomod2nix.toml

Misc

• patterns
• imges
• Alma.md
• NOTES.md
• system.md
• LICENSE
• setup_fabric.bat

Nix code

• flake.nix
• flake.lock
• treefmt.nix
• shell.nix
• pkgs

Docker

• Dockerfile
• docker-compose.yml
• ENV

Proposal

Let's first deal with items unrelated to go that must stay in the root due to technical reasons. They are:

• README.md
• LICENSE
• .devcontainer
• .github
• .git
• .gitignore
• .envrc
• flake.nix
• flake.lock

Next are go related files that I would like to keep in the project root:

• version.go (because it is generated and is more of a parameter rather than application code and doesn't belong in internal where all other fabric code will be moved)
• go.mod
• go.sum
• gomod2nix.toml
• main.go

Reasons for this decision are:

• Familiar workflow for go developers, just execute go build or go run in the project root and it just works
• URL compatibility, some user scripts might have commands like go install https://github.com/danielmiessler/fabric in them and if the go module root moves they are likely to break.
• Moving the go module ROOT into a subdirectory does not give any benefits.

Nix code that is not a flake.nix can be moved around freely so files treefmt.nix and shell.nix and the pkgs directory can be moved to a directory called nix if having them in the root is a problem.

Go code that is not main.go can be moved to a subdirectory called internal or pkg. Their names are based on go-standards. I suggest reading this for yourself but in short pkg is a community convention for parts of the application that can be reused in other projects and internal is a directory respected by the go compiler and won't let others import parts of your application directly. From what I have seen fabric has no externally usable parts, so I will throw everything in internal for now.

Sotring tracking

internal

• core
• cli
• common
• plugins

There are only 3 docker related files excluding .dockerignore. docker-compose.yml and Dockerfile have a reason to be there, developers who use docker probably also prefer to execute their commands from the project root. Moving them is unnecessary if we move files that indeed have no reason to be in the root. The ENV file used by Dockerfile probably can be renamed to .ENV to make it hidden. I am not sure about this because I never provided a prebuilt docker image but moving the docker root might break some docker image builders.

Files that are in the root without any reason that I know of are:

• Alma.md
• NOTES.md
• system.md

I don't know what to do with them, will move to misc directory for now

I guess setup_fabric.bat is in the root for windows users to double click on it with their mouse.(nobody stops them from installing fabric using nix under WSL though). I can only imagine moving this bat script to something like a scripts directory which is useless for now because we don't have any other scripts to move.

images directory can be renamed to .imges or moved to .github

This will reduce the number of items in the root directory. Not by much but enough to not be over the limit of what is acceptable.

And I feel like the real problem is not having a lot of files in the root, but the fact that fabric's internal go packages are in the root and they look the same as other directories like patterns which makes it hard to navigate go code only when you want to ignore everything else and focus on the program itself.

[jaredmontoya]

I implemented these changes on this branch: https://github.com/jaredmontoya/fabric/tree/redesign-project-tree

[eugeis]

Hi @jaredmontoya ,

I like your suggestions and explanations.

@danielmiessler is this reduction enough for you?

I would like to have more reduction in root folder and separation of things like it is in mono repo concept.

Therefore, I don't see big problem to have all Go code in go sub folder. go install will also work, just habe additional go siffix, go install .../go@latest.

All installation and other helfers are typically integration (integ) layer. O would say, there will be come more of such helpers.

[jaredmontoya]

@eugeis can you tell me more about why this project is transitioning to a mono repo?

From what I know mono repos are only useful when there are multiple separate projects(maybe even in different languages) that are two parts of one whole and therefore they benefit from being tracked together under one git history.

An example of such a scenario would be a monorepo where backend is in the application-backend directory and frontend is in the application-frontend directory. And in the root there is usually a docker-compose.yml file to start those two services and provide communication between them. In this scenario a mono repo is useful because the API provided by the backend and API understood by the client are in sync under every commit.

Another example is if the project is really big and has a micro-service architecture, then there are a lot more independent projects in subdirectories all usually managed by docker-compose.yml or some other tool.

But currently fabric isn't one of those scenarios.

Also this is exactly what I meant, If someone made a script that executes go install to install fabric their script will break and they will have to append the required suffix. Not a colossal problem but such breakages may be annoying if done frequently.

If the only reason for this transition to a "monorepo" structure is reduction of items in the root of the project and my proposed changes are not enough, then I can create an integ folder and put the nix directory there and then move docker related files under integ

[eugeis]

@jaredmontoya I received a request from @danielmiessler to reduce the number of files and folders in the root of the repository. You’re correct that our current structure doesn’t have multiple layers like a typical monorepo. However, I do see at least two distinctions—CLI and integration—that could help organize things.

We’re not fully transitioning to a monorepo structure; that was just a comparison to illustrate that moving all Go code into a dedicated go/ subfolder wouldn’t pose any major issues.

Regarding your suggestion to move all Go packages to an internal package, I think that could be misleading to Go developers, as it implies restricted access for internal use only. Instead, for clarity and to reduce clutter, we could create a go/ subfolder to house all packages and related files with a clear, semantic structure.

[jaredmontoya]

@eugeis I am not that familiar with fabric's go codebase so I didn't know that fabric's packages are reusable, I do not insist on the grouping directory being named internal, it might as well be pkg or go.

If you really want to move the GO project root(main.go along with version.go and go.mod+go.sum) to the go subdirectory inside of the project root I won't be radically against it because the only things that I have a strong opinion about are flake.nix, flake.lock and .envrc. They are required to be in the project root due to technical resons.

It is also possible to move gomod2nix.toml around so I can move it to the nix directory away from go.mod if needed.

[eugeis]

@jaredmontoya please move as much as possible to integ/nix.

[jaredmontoya]

@eugeis when fabric is built under a subdirectory called go the resulting binary produced by go build is now called go instead of fabric which is expected behaviour but shouldn't be like this in my opinion.

Moving the go project root out of the repo root creates inconveniences that require use of workarounds like go build -o fabric.

It's time I say what I think after multiple iterations on this feature request.

• The first proposed directory structure in this thread only causes major inconveniences while gains that it provides are easily achievable without moving the go root If fabric doesn't plan to do anything special with the new directory structure in the future.
• Only changes that I came up with do not sacrifice convenience while still improving developer expirience.

Therefore I will do a final iteration based on all the previous iterations that includes only directory structure changes that give gains while taking nothing away.

After that @danielmiessler is free to either merge my final changes if the number of files in the project root turns out to be small enough for for his liking. If he won't be satisfied with that then I will have to wait until the restructuring is complete and then try to fix all the nix features if it is possible.

changes newly included in the final iteration:

• Fixed docker image go version(noticed it was broken while trying to move docker files to "integ", then found out it's also a pain to use docker like that so that's not happening)
• gomod2nix moved to the nix fabric package directory
• added fabric go binary to .gitignore
• moved NOTES.md to misc
• fabric packages are now grouped in pkg so it's easy to focus on fabric's internals and not pay attention to files in the root.

https://github.com/jaredmontoya/fabric/tree/redesign-project-tree

[ksylvan]

@jaredmontoya Where is the strategies folder? It needs to be on the top level as well.

Also, we need to have the two other binaries under their own directories. (to_pdf and code_helper).

I see to_pdf but I don't see code_helper.

I suspect these anomalies are explained by the fact that the branch that you based this reorganization on is very old.

[ksylvan]

@eugeis @jaredmontoya @danielmiessler If you are okay with this approach, I can prepare a PR for the proposal in the next few days.

Project Reorganization Plan

---

Here is an analysis of the current structure and a proposed refactoring plan that aligns with standard conventions for Go project layout.

Analysis of the Current Structure

The current project structure is a mix of Go application code, a Svelte web frontend, Python scripts, Nix configurations, and various other assets at the top level. While functional, this leads to several issues:

• Top-Level Clutter: Numerous files and directories at the root make it hard to quickly understand the project's primary purpose and entry points. Files like main.go, version.go, streamlit.py, and setup_fabric.bat compete for attention with core directories.
• Non-Idiomatic Go Packages:
  • The Go source code is spread across several directories (cli, common, core, plugins, restapi) at the top level. In modern Go, core application code is typically placed within an internal directory to prevent it from being imported by other projects.
  • A common package is often a sign of poor package cohesion, leading to a "grab bag" of unrelated utilities that can create broad, implicit dependencies.
  • A restapi package name describes an implementation detail (REST API) rather than the domain it serves (e.g., server, http, or api).
• Mixed Concerns: The root directory contains application code (Go), web frontend code (web), data processing scripts (Python in Pattern_Descriptions), and configuration (nix, Dockerfile).

---

Proposed Restructure Plan

The following structure organizes the project by concern, adheres to Go community standards, and clarifies the boundaries between the different parts of the application.

Proposed New File Structure

Here is the high-level view of the proposed structure. Unchanged directories like web and nix are kept as-is but shown for context.

.
├── cmd
│   └── fabric
│       └── main.go              # Main application entrypoint
├── internal
│   ├── cli                      # All CLI-related code
│   ├── core                     # Core application logic (e.g., chatter)
│   ├── domain                   # Domain types, moved from 'common'
│   ├── patterns                 # Logic for loading/managing patterns
│   ├── plugins                  # All plugin logic (ai, db, etc.)
│   ├── server                   # The 'restapi' code, renamed
│   └── util                     # Specific, shared utilities (use sparingly)
├── data
│   ├── patterns/                # All pattern markdown files
│   └── strategies/              # All strategy json files
├── scripts
│   ├── analysis
│   │   ├── streamlit.py
│   │   └── requirements.txt
│   └── pattern_generation
│       ├── extract_patterns.py
│       └── ...
├── web/                         # (Svelte frontend, unchanged)
├── docs
│   ├── images/
│   ├── NOTES.md
│   └── Pattern_Descriptions/    # Documentation about patterns
├── completions/                 # (Shell completions, unchanged)
├── nix/                         # (Nix environment, unchanged)
├── Dockerfile
├── docker-compose.yml
├── go.mod
├── go.sum
├── LICENSE
├── README.md
└── version.go                   # Can stay here or move to internal/version

---

Rationale for Key Changes

Here’s a breakdown of why these changes are recommended:

1. Introduction of the cmd/ Directory

• What: All main application entry points are moved into cmd/. The primary executable's code moves to cmd/fabric/main.go.
• Why: This is a standard convention for Go applications that have a clear separation between reusable libraries and executable commands. It immediately tells a new developer where the program starts. It also allows for the creation of other commands in the future (e.g., cmd/fabric-server/) without polluting the root.

2. Introduction of the internal/ Directory

• What: The majority of the Go packages (cli, core, plugins, restapi, common) are moved under internal/.
• Why: The Go toolchain enforces a special rule for the internal directory: code within it can only be imported by code inside the same parent directory. This makes your application's core logic private. It prevents other external Go projects from importing your application-specific code, which is not designed to be a public library. This clarifies the project's public API (which in this case, is zero) and prevents unintentional dependencies.

3. Reorganizing and Renaming Packages Inside internal/

• restapi -> internal/server: The package is renamed to reflect what it does (provides an HTTP server) rather than how it does it (with a REST API). This is a more abstract and stable name.
• Dissolving common: The common package is broken up.
  • domain.go moves to internal/domain. This gives your core business objects a clear, dedicated home.
  • utils.go would ideally be eliminated. Its functions should be moved closer to the packages that use them. If some are truly shared by all packages, they can live in a well-defined internal/util package, but this should be avoided if possible.
• patterns logic: Code for loading and managing patterns from the filesystem (like patterns_loader.go) should be consolidated into a dedicated package, for example internal/patterns.

4. Consolidating Data and Scripts (data/, scripts/)

• data/: The patterns/ and strategies/ directories contain data assets consumed by the application, not source code. Moving them into a data/ directory cleans up the root and signals that these are passive assets. The application would then be configured to look for them in data/patterns.
• scripts/: The streamlit.py, requirements.txt, and the Python scripts from Pattern_Descriptions are helper scripts, not part of the core Go application or the web frontend. Grouping them under a scripts/ directory makes their role clear. The non-idiomatic Pattern_Descriptions directory is renamed and moved.

5. Cleaning the Root Directory

• After these moves, the root directory becomes much cleaner, containing only project-level configuration (go.mod, Dockerfile, nix/), documentation (README.md, LICENSE), the main application modules (cmd, internal), and the self-contained web frontend (web).
• version.go can either remain at the root (as it's often set at build time via linker flags) or be moved into an internal/version package for better organization.

Step-by-Step Migration Plan

1. Create New Directories: Create the cmd/fabric, internal, data, scripts, and docs directories.
2. Move Go Packages: Move the cli, core, plugins, and restapi directories into internal/. Rename internal/restapi to internal/server.
3. Refactor common: Move the contents of common into more appropriate packages (e.g., internal/domain, or into packages that use the specific utility functions).
4. Move main.go: Move main.go to cmd/fabric/main.go.
5. Update Imports: This is the most significant step. Use your IDE's refactoring tools or a tool like gofmt or goimports to update all import paths in your .go files. For example, an import of "github.com/danielmiessler/fabric/cli" would become "github.com/danielmiessler/fabric/internal/cli".
6. Move Data Assets: Move the patterns/ and strategies/ directories into data/. Update the application configuration to read from these new paths.
7. Move Scripts: Move the various Python and shell scripts into the scripts/ directory.
8. Test Everything: Run go build ./cmd/fabric, execute your tests with go test ./..., and run the application to ensure that the CLI, API, and pattern loading all function correctly after the refactor.

[eugeis]

Hi @ksylvan, Thanks for putting the plan together.

I suggest moving all the Go code into a dedicated folder—either named fabric or preferably just fa—so that the resulting CLI binary is simply named fa.

[ksylvan]

I suggest moving all the Go code into a dedicated folder—either named fabric or preferably just fa—so that the resulting CLI binary is simply named fa.

Hmmm... A binary named fa? I'm not quite sure I follow the logic. Is that supposed to be a short version of fabric-ai?

[jaredmontoya]

@ksylvan Your file structure representation looks great and it's similar to what I was trying to do. Don't forget to update the Update Version File and Create Tag github action according to your changes.

The only weird thing I noticed is that in your representation there are no flake.nix and flake.lock in the root directory but I don't see mentions of them being moved anywhere so I assume that this is just a mistake.

I don't think we should move ALL go code under one dedicated folder like @eugeis suggests and your approach is better in my opinion.

Also I think that @eugeis means that the fabric binary that is built when you call go build will become fa if the folder is named fa and a shorter binary name is easier to type on the command line. But that can be changed later after the restructuring is done by renaming cmd/fabric to cmd/fa.

[ksylvan]

Okay, I get that. @jaredmontoya @eugeis

I'm almost always typing f a b [TAB] (followed by, for example, -p summar [TAB])

Or even: summari {TAB] and then -m {start of model name} [TAB] since the aliases in zsh are still able to use the Fabric command completions.

I'm going to leave the fabric executable name alone for now, and, like you said, we can revisit that later. I suppose people can also add alias fa=fabric (or alias fa=fabric-ai if they are using the Homebrew package).

That's one other thing: I'll need to make sure the brew package will keep building correctly after the reorganization.

[jaredmontoya]

I also think people can just alias fa to fabric and it is not necessary to change the binary name.

[ksylvan]

I'm about to start doing this at @danielmiessler's request.