Fixing the lock file

[janhartigan]

Hey @franzliedke thanks a lot for taking the time to put together the recent release.

However, I have noticed one significant issue with the newly-generated composer.lock file. Normally composer generates something like this:

{
    "name": "owner/repo",
    "version": "dev-master",
    "source": {
        "type": "git",
        "url": "[email protected]:owner/repo.git",
        "reference": "2cf2e58fc8b71f44dbcca306cf7e16a65a2d4e97"
    },
    ...
}

There would also be a dist with the zipped contents of the git repo.

With the new updates, the composer lock file now looks like this:

{
    "name": "owner/repo",
    "version": "dev-master",
    "dist": {
        "type": "path",
        "url": "../../path/to/repo/",
        "reference": "2cf2e58fc8b71f44dbcca306cf7e16a65a2d4e97",
        "shasum": null
    },
    ...
}

Trouble with this is that when we push the code up through the pipeline, our servers will try to install that composer.lock file and it won't be able to figure out how to grab the dist. Since you're already using this in a real project, I'm guessing there's some other approach that I'm missing. Let me know what you think.

[franzliedke]

Darn, that really is a problem I did not foresee. I probably did not notice this because I did not commit the lock file (yet) in the projects where I was using this.

I'll have to think about that one for a bit...

[tobyzerner]

I'm having a hard time thinking of a clean way to fix this. Should the onus be on the user to temporarily disable studio and then run composer update if they want to commit an updated lock file? Maybe Studio could automate this process (studio lock or something)

[RemiCollin]

I guess ideally Studio would hook in right after composer.lock file is created, right before generating autoload files. I don't have enough knowledge of the inner workings of composer to say if that's possible or not, though.

[franzliedke]

I haven't looked into the implementation yet, but if the entries in composer.lock are generated by the repository instances that Composer uses, we might be able to fix this by using our own repository implementation, as hinted at in #56.

[janhartigan]

@franzliedke that sounds like a great idea. So everyone is aware, these are the repositories in question:

https://github.com/composer/composer/tree/master/src/Composer/Repository

Looks like the default version is the ComposerRepository. The one that Studio is currently using is the PathRepository.

Man, looking at that ComposerRepository, that library desperately needs some refactoring work.

Edit: Actually I guess we have to be careful here as we should fall back to the repository that would it have used if Studio didn't exist. That might be something like a VCS repository (as is the case in our system).

[jonnywilliamson]

I'm having this exact same issue. I can't commit my lock file at the moment.

I have no suggestions as I don't have enough programming skill to come up with a solution, but thanks for looking into this.

[mnpenner]

So...until this is fixed, we have to manually edit studio.json, clear out the paths, run composer update on each of the repos, commit/push and then restore the paths?

studio scrap sounds like it wants to delete my whole project, I just wanted to temporarily unlink it.

[LavaToaster]

@mnpenner Out of curiosity, if you ran composer update --no-plugins does that work?

[janhartigan]

@mnpenner alternatively you can downgrade to an older version of studio (0.9.5). That's what we've done until it's fixed.

[warksit]

@Lavoaster yes totally, Have changed install scripts on production servers to composer update --no-plugins (and other flags such as --no-dev) and all good.

[janhartigan]

@warksit trouble with that is that composer update wasn't really intended to be run in a production environment.

[swt83]

I can't use version 0.9.5 bc Composer gives me some error:

  Problem 1
    - Installation request for franzl/studio 0.9.5 -> satisfiable by franzl/studio[0.9.5].
    - franzl/studio 0.9.5 requires composer-plugin-api 1.0.0 -> no matching package found.

The newer versions of this repo (anything post 0.9.5) don't seem to actually load the packages referenced in my studio.json file. So now I'm stuck.

[janhartigan]

@swt83 unfortunately to continue using 0.9.5, you need to have version 1.0.0 of composer installed. There isn't really a good way around this at the moment.

[swt83]

Why am I not able to get studio to work with version 0.11.0? The studio.json file is correct, but the code isn't actually loading into my app.

[janhartigan]

@swt83 so you're running the latest composer version and you've successfully run composer global require franzl/studio?

[swt83]

I'm not using the global method. I have the studio package installed on the project and I have the studio.json file setup, and when I run composer update it lists off all the studio packages it is loading -- but then the packages don't actually become available to my project.

[warksit]

@janhartigan you're right. However --no-plugins works fine with composer install.

[franzliedke]

I'll get back to you later today, sorry for the radio silence lately, folks.

[swt83]

I think you guys know all this stuff already, but here is my report:

I ended up removing Studio from my project and manually setting up a path type repository in my composer.json file. I used a wildcard * to have it load my entire packages directory, which is where I am working with all my private repos.

"repositories": [
        {
            "type": "path",
            "url": "workbench/*/*"
        }
],

I had to also manually add all my packages VCS dependencies to the master composer.json for my project because Composer won't recursively load VCS dependencies.

This was all a massive pain but it's got me up and running again.

For whatever reason Studio says it's loading packages when I run composer update but those packages just don't work. The old 0.9.5 version had kept me going for a few months, but now that won't work anymore due to Composer updates. So now I'm left with manually setting up the composer.json file as described above.

[janhartigan]

@warksit hm strange. As I understand it, composer install merely tries to build a project from a composer.lock file. If a lock file has this:

{
    "name": "owner/repo",
    "version": "dev-master",
    "dist": {
        "type": "path",
        "url": "../../path/to/repo/",
        "reference": "2cf2e58fc8b71f44dbcca306cf7e16a65a2d4e97",
        "shasum": null
    },
    ...
}

How could it know where to get the repo, if not using the path method on the local file system? I haven't tested this locally as I'm still using 0.9.5, but you may want to double check that.

@swt83 hm must be a quirk of the non-global version. In the global version you just need to install it and have a studio.json file next to your composer.json file. I'm guessing that @franzliedke has some ideas up his sleeve.

Also...hi @franzliedke! No worries about the radio silence, if I really wanted to get anything done, I should just fix it myself and create a PR.

[franzliedke]

@swt83 The structure of studio.json changed in v0.11.0. So that probably explains why things did not work anymore after the upgrade. Although technically I built things in such a way that it should fall back to the old parser, so that is confusing.

Can you post your studio.json, please?

Also, if you run studio create ... it should rewrite the file to the new structure. (You can then use studio scrap to get rid of the temporarily created package.)

P.S.: Please open a new issue about this, it seems to be unrelated.

[rask]

Any news on this? Currently tiptoeing around the composer.lock related issue while sharing projects to teammates. Not a huge issue right now, but would like to see this streamlined a bit. :)

[laurencei]

Hi everyone,

So I started using this package, but ran into this issue with composer.lock.

I ended up removing the studio package - but using the core concept in a different way. Rather than trying to modify .lock files and change the inner workings of composer - I think it is easier to let composer do its own thing - and just hook in before and after.

The concept is change the script to do;

1. Remove the symlink file
2. Run composer update (let it run normally)
3. Remove the package folder
4. Re-symlink

This means composer has no idea and is completely unaffected by any local development. The only thing that will occur during step 2 is composer will detect your package is not actually in the vendor file, and re-download the current "latest" from packagist - but that is unlikely to be a big issue (especially as it'll probably be cached anyway).

I've started using this for me - and it is working perfectly. The composer.lock is completely unaffected and is totally unaware of what is going on.

Maybe this concept could be used in this package? You could register each package as you currently do with studio - then have it run hooks before and after to perform the commands as needed for each package rather than during the update/install;

Here are the commands I am using:

rm /full/path/to/your/project/vendor/your-vendor-name/your-package-name
composer update
rm /full/path/to/your/project/vendor/your-vendor-name/your-package-name -rf
ln -s /full/path/to/your/project/vendor/your-vendor-name/your-package-name /full/path/to/local/your-package-name

[franzliedke]

Hi @lioannou!

I agree this is the best way forward. So, basically, Studio would always load all required projects via symlink, if it can find them locally, doing so after Composer does its usual thing and resolves the packages from Packagist etc. (this would probably still need the local repository as fallback, for when the package hasn't yet been added to Packagist).

This leaves the question of what happens when a package is loaded via symlink - just because it is there -, despite not matching the version constraint defined in composer.json. Is it okay if we simply show a warning in that case?

[laurencei]

this would probably still need the local repository as fallback, for when the package hasn't yet been added to Packagist

It could be a private VCS etc though. And people who want to develop a new package can create a local version otherwise.

This leaves the question of what happens when a package is loaded via symlink - just because it is there -, despite not matching the version constraint defined in composer.json. Is it okay if we simply show a warning in that case?

Is it any different to what Studio already does though? You could do this:

> composer update
Studio: Hey - I'm removing symlinks for Package 1, Package 2 and Package 3
composer ... updating... beep... boop.... done...
Studio: Hey - now that Composer is finished - I've resymlinked Package 1, Package 2 and Package 3 to your local development version (this may be different from your composer.lock file)

That makes it visually explicit what packages have had there symlinks removed and/or re-add?

[janhartigan]

This is actually the implementation that we've already done in our system. There are some issues with permissions (especially in Windows) if you use PHP's symlink. The scripts also handle the scenario where you try linking and it's already linked, and if you try unlinking when it's already unlinked. In the end we saw the fewest issues when just building the exec commands manually. You guys can use this if you find it useful.

Here's are the scripts:

<?php namespace App\Console\Commands;

use Illuminate\Console\Command;

class LinkDevelopmentCode extends Command {

	/**
	 * The console command name
	 *
	 * @var string
	 */
	protected $name = 'api:link-dev-code';

	/**
	 * The console command description
	 *
	 * @var string
	 */
	protected $description = 'Links the development code';

	/**
	 * Execute the console command
	 */
	public function fire()
	{
		$targetPath = base_path('vendor' . DIRECTORY_SEPARATOR . 'path' . DIRECTORY_SEPARATOR . 'to' . DIRECTORY_SEPARATOR . 'api');

		if (!$apiDevPath = env('API_DEVELOPMENT_PATH'))
			return;

		if (!is_dir($apiDevPath))
			return $this->error("Could not find the API development path at {$apiDevPath}.");

		//if this is a sym link, delete it
		if (file_exists($targetPath) && $this->pathIsLink($targetPath))
		{
			exec($this->buildDeleteLinkCommand($targetPath));
			$this->info("Existing symlink deleted");
		}
		//if this is a directory move it to its stored location
		else if (is_dir($targetPath))
		{
			exec($this->buildMoveDirectoryCommand($targetPath));
			$this->info("Original vendor path/to/api directory stowed in the closet");
		}

		exec($this->buildLinkCommand($apiDevPath, $targetPath));

		$this->info("The symlink was created!");
	}

	/**
	 * Determines if the provided path is a symbolic link
	 *
	 * @param  string    $path
	 *
	 * @return bool
	 */
	protected function pathIsLink($path)
	{
		return is_link($path) || (array_diff(stat($path), lstat($path)));
	}

	/**
	 * Builds the OS-dependent symbolic linking command
	 *
	 * @param  string    $apiDevPath
	 * @param  string    $targetPath
	 *
	 * @return string
	 */
	protected function buildLinkCommand($apiDevPath, $targetPath)
	{
		if ($this->isWindows())
			return "mklink /J {$targetPath} {$apiDevPath}";

		return "ln -s {$apiDevPath} {$targetPath}";
	}

	/**
	 * Builds the OS-dependent link deletion command
	 *
	 * @param  string    $targetPath
	 *
	 * @return string
	 */
	protected function buildDeleteLinkCommand($targetPath)
	{
		if ($this->isWindows())
			return "rd {$targetPath}";

		return "rm {$targetPath}";
	}

	/**
	 * Builds the OS-dependent directory move command
	 *
	 * @param  string    $targetPath
	 *
	 * @return string
	 */
	protected function buildMoveDirectoryCommand($targetPath)
	{
		$newTargetPath = $targetPath . 'installed';

		if ($this->isWindows())
			return "move \"{$targetPath}\" \"{$newTargetPath}\"";

		return "mv {$targetPath} {$newTargetPath}";
	}

	/**
	 * Determines if the current OS is Windows
	 *
	 * @return bool
	 */
	protected function isWindows()
	{
		return strpos(php_uname(), 'Windows') !== false;
	}

}

And here's the unlinking command:

<?php namespace App\Console\Commands;

use Illuminate\Console\Command;

class UnlinkDevelopmentCodeCommand extends Command {

	/**
	 * The console command name
	 *
	 * @var string
	 */
	protected $name = 'api:unlink-dev-code';

	/**
	 * The console command description
	 *
	 * @var string
	 */
	protected $description = 'Unlinks the development API code';

	/**
	 * Execute the console command
	 */
	public function fire()
	{
		$vendorPath = base_path('vendor' . DIRECTORY_SEPARATOR . 'path' . DIRECTORY_SEPARATOR . 'to' . DIRECTORY_SEPARATOR . 'api');

		//if this is a sym link, delete it
		if (file_exists($vendorPath) && $this->pathIsLink($vendorPath))
		{
			exec($this->buildDeleteLinkCommand($vendorPath));

			$this->info("Existing API symlink removed");
		}

		//if the stored vendor path already exists, move it into its original location
		if (file_exists($this->buildStoredVendorPath($vendorPath)))
		{
			exec($this->buildMoveDirectoryCommand($vendorPath));

			$this->info("Original Composer API installation restored");
		}
	}

	/**
	 * Determines if the provided path is a symbolic link
	 *
	 * @param  string    $path
	 *
	 * @return bool
	 */
	protected function pathIsLink($path)
	{
		return is_link($path) || (array_diff(stat($path), lstat($path)));
	}

	/**
	 * Builds the OS-dependent directory move command
	 *
	 * @param  string    $vendorPath
	 *
	 * @return string
	 */
	protected function buildMoveDirectoryCommand($vendorPath)
	{
		$storedVendorPath = $this->buildStoredVendorPath($vendorPath);

		if ($this->isWindows())
			return "move {$storedVendorPath} {$vendorPath}";

		return "mv {$storedVendorPath} {$vendorPath}";
	}

	/**
	 * Builds the OS-dependent link deletion command
	 *
	 * @param  string    $vendorPath
	 *
	 * @return string
	 */
	protected function buildDeleteLinkCommand($vendorPath)
	{
		if ($this->isWindows())
			return "rd {$vendorPath}";

		return "rm {$vendorPath}";
	}

	/**
	 * Builds the vendor path location where the composer api installation is stored
	 *
	 * @param  string    $vendorPath
	 *
	 * @return string
	 */
	protected function buildStoredVendorPath($vendorPath)
	{
		return $vendorPath . 'installed';
	}

	/**
	 * Determines if the current OS is Windows
	 *
	 * @return bool
	 */
	protected function isWindows()
	{
		return strpos(php_uname(), 'Windows') !== false;
	}

}

[AndrewFeeney]

Any word on this? If there's a way it can be done I'd love to help with a PR. We have a ship terminal command in github.com/arcframework/framework which currently runs composer update so those of us running a local dev build of Arc Framework don't end up with a broken plugin build, but I'd love to contribute a fix so we can switch that to composer install.

[franzliedke]

Whew, that took a while. Another vacation passed. Sorry, guys.

I worked on this, and I think I found a solution. Please try out the newly released 0.14.0-beta1 version. I am hoping this issue will be gone.

(Note that you will have to run composer update and composer dump-autoload for the installation of any package.)

I am very tired right now, so let me just quote the commit message here:

This is a fundamental rewrite of the Composer integration. Now, instead of adding the loaded paths to Composer's search path (by creating path repositories for them), we replace the packages downloaded by Composer that can be found in the loaded paths by symlinks to the local paths.

Doing so requires us to hook into the autoload dumper, which now has to respect the rules in the local path, not those obtained from Packagist.

All of this should hopefully fix several issues, most importantly:

• Composer's lock file will be written before Studio does its magic, therefore not causing any conflicts with other developers' setups.
• Different version constraints on symlinked packages won't cause problems anymore. Any required packages that are found in loaded paths will be loaded, no matter the branch or version they are on.

Open questions:

• How should packages be handled that have not yet been added to Packagist? (Proposed solution: Create path repositories for the loaded paths, but append them instead of prepending, so that they will only be used as fallback, if Packagist does not yield any results.)
• Should we validate the constraints from composer.json before creating symlinks? With this setup, everything might be working locally, but not when downloading the package from Packagist (as another version may be downloaded instead).

Looking forward to your reports and your input. :smile:

[franzliedke]

Any luck anyone? A simple "Seems to work" is welcome, too.

And yes, the irony of me rushing all of you after taking so long myself is not lost on me. In case someone wondered. :wink:

[nathanmerrill]

I can't get it to work (but it may be my fault, I'm unsure). Here's what I've done:

I removed the composer.lock file and the symlinked folder from /vendor. I ran composer dump-autoload and then composer update, which gives me:

The requested package my/package could not be found in any version