RFC: Cloud Deployments, State Storage & Registry

[austencollins]

Hi all,

We've been hard at work on Serverless Components and we're looking for feedback on a few large and powerful updates we've implemented in a special branch of Serverless Components.

The changes are listed in detail below. If you want to try them out, go to the cloud branch of the Serverless Components project and check out the Quick Start.

Please leave feedback and questions in this issue and thank you in advance for checking this out. It's taken a lot of work to pull these off. We hope you enjoy!

Our Goals

To clarify why we are considering these updates, we've listed our overall goals for Serverless Components below, as well as a status update on how well we accomplished them with the first implementation of Serverless Components...

Goal 1: Enable new abstractions for all serverless services and use-cases

The first goal of Serverless Components is to offer new abstractions, that enable developers to provision more serverless cloud infrastructure and more use-cases, more easily. We also want every developer to be able to make them.

Every Serverless Component is an abstraction, designed around a use-case or pattern, that offers its own simple configuration syntax in serverless.yml and uses that syntax to provision serverless cloud infrastructure behind the scenes, to deliver a serverless outcome, without developers having to do that themselves.

So far, Serverless Components has achieved this extremely well. It has enabled developers to build tons of new serverless use-cases, like websites, Next.js apps in the edge, websockets APIs, CDNs, and much more, easily.

Currently, Serverless Component deployments are growing over 50% month-over-month. While 80% of Serverless Components have been built by developers outside of Serverless Inc.

Goal 2: Enable everyone to share their Components

We want every developer and team to be able to easily share their Serverless Components.

We sought to offer this by leveraging the NPM package manager as the Serverless Component Registry. However, this has been only minimally successful.

By relying on NPM, it has created a lot of complexity, and it has been hard to deliver the developer experience we want to achieve while being stuck with NPM’s conventions. Particularly, users have to work with NPM's organizations concept, users are stuck with their namespaces, users have to manage private Components via NPM, users are limited to Javascript, and many more issues have arisen...

Goal 3: Enable instant deployments of Components

The development process is the BIGGEST pain-point for developers when building serverless applications.

Because serverless services exist only on the cloud, developers either have to deploy their projects every time they make a change—which is painfully slow, or emulate cloud services locally on their machines.

While there has been a lot of interesting innovation to make local development easy, with the growing number of serverless cloud services, we believe it is in every team’s best interest to work in the real cloud environment as much as possible.

To help developers work in the cloud, we have been working extremely hard to make Serverless Component deployments as fast as possible.

We’ve made tremendous progress here. By parting ways with AWS CloudFormation, working with service APIs on AWS and others directly, and designing Serverless Components to provision one thing as quickly as possible, most Component deployments happen within 8-12 seconds—a big improvement from AWS CloudFormations 30 second to 5 minute deployments.

Updates

Here are the list of updates we have implemented for feedback, and how they relate to our goals:

Cloud Deployments

When researching how to make Serverless Components deploy even faster than 8-12 seconds, we discovered it was possible if deployments were performed in the cloud.

The result is that Serverless Components are now reaching deployment speeds of 2-4 seconds!

The reason for this is simple. When Components run on a local machine (like your laptop), they have to make many public network requests to interact with AWS APIs. However, when Components run in the cloud, AWS API requests happen significantly faster because they don't have to cross the public network.

Now, there is truly no other serverless development experience that deploys this quickly and enables you to see your updates in the real cloud environment instantly, rather than having to emulate everything locally to achieve the same pace of feedback.

The best way to think about this change overall is to consider Serverless Components a cloud service, like AWS CloudFormation, rather than javascript libraries.

Even better, we believe we can make this even faster...

Serverless Registry

In order to support Cloud Deployments, Serverless Components have to live in the cloud by default. Further, we also wanted to solve our NPM pain-points, and streamline the Serverless Component sharing experience for Component authors.

The only way to pull all of this off was to build our own Serverless Component Registry and part ways with NPM for good. So, we built a Registry, and now anyone can deploy their Components there, to share with others.

When you run serverless publish, your Component code is uploaded to the Registry where it awaits to be used by anyone.

Now anyone can more easily register their Components and make them accessible to everyone else.

This is the absolute easiest way to share serverless use-cases, patterns and applications with your team and the world.**

We will be adding a front-end to the registry within the next 2 weeks, featuring search and more!

Remote State Storage

Over first version of Serverless Components saved state locally (in .serverless), which made it impossible to collaborate on your deployed Component Instances via Github/Gitlab/BitBucket and run your Component Instances in CI/CD. Naturally, this was a major issue.

Now, when you run Components, their state is automatically saved in the cloud.

Just make sure your teammates are collaborators in your Serverless Framework Organization, they can easily work on your Component Instances.

We're also storing Component outputs in the cloud as well. Within a few weeks, we will enable referencing outputs remotely in your serverless.yml from other Component instances.

Stages

Serverless Components now have a Stages concept, which enables you to deploy entirely separate Component Instances and their cloud resources per stage.

The dev Stage is the default. If you wish to change your stage, set it in serverless.yml, like thisL

component: [email protected]
org: my-org
app: my-app
name: my-component-instance
stage: prod # Enter the stage here

You can also specify a Stage upon deployment, which overrides anything set in serverless.yml, like this:

$ serverless deploy --stage prod

Breaking Changes

We believe these updates are huge steps forward for Serverless Components and potentially the overall serverless movement. However, they do result in some breaking changes, which you must be aware of.

Single Instance serverless.yml

After observing user behavior, we have changed serverless.yml to only contain 1 Component instance, like this:

# serverless.yml - An example of a Serverless Component for making a serverless express.js API

component: express # The name of the Component in the Registry
org: acme # Your Serverless Framework Org
app: fullstack # Your Serverless Framework App
name: rest-api # The name of your instance of this Component

inputs: # The configuration the Component accepts according to its docs
  src: ./src

Why did we do this?

The first reason is we believe you should separate your infrastructure resources as much as possible. When combining a lot of cloud resources via multiple components in the same serverless.yml, you can run into slow deployments and riskier deployments. People keep combining resources they deploy often (functions/code) with resources that shouldn't be deployed often (databases).

The second reason is most important. Since we are storing state and outputs of your Component Instances in the cloud now, the ability to reference outputs stored remotely, from another Component instance, is incredibly easy and powerful. It also makes collaboration a breeze. We believe this is the right way to link Components and we'll be building even more features on top of this to help recognize upstream and downstream dependencies. Please note that we have not yet enabled Component Output references in the cloud yet, but they will be available within the next 2 weeks.

src input changes

Component Input & Output Types are something we are starting to invest in.

The first Component Input Type is the src input, which declares a directory containing files/code needed for your Component's provisioned outcome (e.g. code for your website).

The src input looks like this:

inputs:
  src: ./src # Source code directory

or this:

inputs
  src:
    src: ./src # Source code directory
    hook: npm run build # Contains a script to run before deployment
    dist: ./dist #  Directory of the built artifact to use instead of the `src` directory.

When Components deploy, files in the src input are first uploaded, before Deployment.

serverless.js changes

In order to make Serverless Components run in the cloud, we had to make some slight modifications to serverless.js. We also took this opportunity to simplify the APIs further. While these aren't major modifications, your Components won't work without them.

First, you should be aware that serverless.js no longer runs locally—at all. It is designed to only run in the cloud.

These changes are:

• this.context.credentials is now this.credentials
• this.context.status() is now await this.status()
• this.context.debug() is now await this.debug()
• The default() Component method is now simply deploy()
• You don't need to require child components via NPM any more. Since Components live on the cloud now, you can simply initialize them by running this.load() and inputting the Component name + version in the registry.

serverless.js development workflow

Since Components only run in the cloud, the development workflow is a bit different because you have to push to the Serverless Registry before you can test them.

We make this easy by giving you a dev version of your Component you can build and test on. Instructions on how to do this are in the updated README.

Try It Out

We've implemented all of these changes and we're looking for your feedback!

The cloud branch contains all of the changes. It's README has been updated to contain documentation on this new implementation of Serverless Components. Read the Quick-Start on how to get started quickly.

[Pavel910]

@ac360 @eahefnawy my first and most obvious question is about the fact that serverless.yml now only contains a single component. So how would a deployment of, for example, 2 Lambdas and an API Gateway look like?

Does this mean we now need to create some sort of automation to deploy the entire infrastructure, or you suggest that we should put all our infrastructure in 1 component, and use this.load to abstract the moving parts away from the user?

Also, it seems that now everybody MUST have an account on your platform, to be able to do anything. Publishing of components is tied to the cloud, remote state storage, deployment... that's a huge drawback, no matter the speed gains.

Deploying from your cloud is also a problem from the debugging perspective. It means that if something goes wrong, and it does go wrong most of the time during development, I'm unable to go into node_modules and add console.logs in 3rd party components. Moreover, I have to use your cloud UI to read the console.log messages.

I've already read this RFC twice; need to think more about this and I'll come back with more questions.

[austencollins]

Hi @Pavel910,

We were hoping to hear from you. You have a unique and cool use-case for Serverless Components. We actually believe where we're heading with this might help your use-case a lot, though we want to chat more with you about everything. We're wondering if you'd be open to chatting about these and potential future changes. If you're available over the next 1-2 weeks, could you please email me at austen AT serverless.com?

Does this mean we now need to create some sort of automation to deploy the entire infrastructure, or you suggest that we should put all our infrastructure in 1 component, and use this.load to abstract the moving parts away from the user?

Regarding 1 Component per serverless.yml... When it comes to the Functions and API Gateway use-case for building serverless REST APIs, we have a few Components planned that will combine these into one Component, to offer a better experience. We're also seeing users do the same thing.

While combining functions and API Gateways is fine, we don't generally recommend combining a lot of things in the same infrastructure stack. We see users regularly combine databases, VPCs, domain configs, with functions/code, and running into issues with many of those resources because they are deploying their stack often to update their code, and occasionally, a bug affects their other resources. That situation is very hard to come back from...

So, we went down the path of single Component per serverless.yml. Given it's easy to author Components that provision multiple things, it made more sense to do the single Component per serverless.yml approach. Further, if anything knows how to best provision multiple resources safely and quickly, it's going to be a parent component.

Would love to chat with you more about this though..

Also, it seems that now everybody MUST have an account on your platform, to be able to do anything. Publishing of components is tied to the cloud, remote state storage, deployment... that's a huge drawback, no matter the speed gains.

We definitely know how much users don't like signing up for things. But we are about to offer a ton of ridiculously cool dev tooling on top of this, that incurs cost and could be abused. Further, we need to associate their state with something, and enable them to share outputs with others. For these reasons, and future reasons, we went down this path. It would be helpful to hear more about why you believe this is a drawback in our chat.

Deploying from your cloud is also a problem from the debugging perspective. It means that if something goes wrong, and it does go wrong most of the time during development, I'm unable to go into node_modules and add console.logs in 3rd party components.

Yes, this is the serverless development issue in a nutshell. We hadn't though about the use-case of wanting to put console.logs in Components authored by others. However, all this.debug statements, from the parent and child Components, are streamed to the CLI in real-time from the cloud via websockets. So you can actually see the this.debug statements to get a better sense of what's going wrong if a child Component is affected. This real-time streaming from the cloud will be the back-bone of some killer features in the next few weeks. We'll share our plans here when we chat, and see if those features can cover your concerns.

Moreover, I have to use your cloud UI to read the console.log messages.

While the debugging features there are rapidly improving, we don't really want people to have to use the UI during development and are trying to make the UI not an integral part of the process.

[eahefnawy]

Adding to what @ac360 have already said, I think it's important to really think about your own use case (ie. why do you need couple of lambdas and an APIG). Based on this, you could write your own component that drives your own use case better than any other available component. You are of course welcome to use any component in the registry as a child, but it goes down to whether this would end up being the best experience for your own use case.

[Pavel910]

@ac360 email sent! Thanks for the opportunity to chat :)

I just wanted to post a reference to one of our serverless.yml files: https://github.com/webiny/webiny-js/blob/master/examples/api/serverless.yml

Currently this is how we deploy things, and it works fine. YAML is not the best way to build infrastructure definitions but it is ok for now. What I really like about this, is the readability and ease of configuration. We can easily add more services, more AWS resources, etc, cross-component dependencies; it's easy to remove stuff and easy to add stuff. We've developed a way to only deploy single components so we don't deploy the entire stack all the time.

One thing we use a lot is code generation. Our components often generate boilerplate code, bundle it with webpack and deploy the bundle. This is something that will give lambda some headaches I believe (during the deploy process).

Since we are a development platform ourselves, we're really trying our best to reduce the amount of commands a developer needs to run in order to deploy everything. Splitting this config file into separate files will not help us in that. If, on the other hand, we hide everything in a single component, we will have dozens of inputs which is also not really helpful.

Looking forward to discussing this :)

[Pavel910]

@eahefnawy one other thing I would strongly recommend using is the debug library for logging debug statements. We're currently monkey patching the debug function like this component.context.instance.debug = debug(`webiny:${alias}`);. That way we can clearly see the output of each component and we can see how much time passed between debug logs, thus making it easier to find out which parts of deploy process take significant amount of time.

Here is an example output:

As you can see this makes it much more readable and you can easily highlight important information in your output.

[eahefnawy]

That IS beautiful @Pavel910 ... I'll try that 🙌

[barrysteyn]

Some thoughts:

1. Single instance in serverless.yml is such a major change that is basically means all the work that people have done up to now will be thrown away. It's that big. Myself (and many other people) use serverless nextjs component - with this change, most of that work will not work!
2. Registry: NPM is great for JavaScript. I understand that perhaps it may not suit other language needs. However, since I suspect that JS is probably the main language that will be used, why can't we stick to JS by default and use another registry for other languages? Besides the language issue, I don't understand the issues that NPM poses: It is popular and works fine for most people.
3. Cloud deploy is a cool idea, but what if I don't care about speed and want to deploy on my local machine (for security reasons). Couldn't cloud deploy be an optional extra? If it is a money thing, could there be a paid version that allows for one to deploy locally.

Interesting to hear back.

[danielcondemarin]

Here are a few thoughts from me:

• It would be great if you could expand more how Cloud Deployments work behind the scenes. This will enable component developers to have a better understanding of the benefits or more importantly any limitations. A blog post, architecture diagram on this would be ace.

• How does this work for components like https://github.com/serverless-components/aws-s3 ? Unless I misunderstood something this type of component doesn't seem to fit into the Cloud Deployment model.

• What if my component needs to read files (like assets, images etc. not source code) during deploy time? Presumably these files are only available in the local filesystem and wouldn't be available in the cloud.

I'd love to see how https://github.com/danielcondemarin/serverless-next.js fits into this new model. However, will wait to hear a bit more from this thread before having a go 🚀

[austencollins]

Hi @barrysteyn,

We'd love to chat with you about this to hear more from you and show you more of what we're considering. If you have some time next week, could you send me an email, so we can set something up? I'm austen AT serverless.com

If we do a call in the morning, perhaps we can have the amazing @danielcondemarin join us as well (since he's in the UK).

[Pavel910]

@ac360 Any updates on this? Would love to know the general direction of this RFC. Thanks!

[KowalskiTom]

1. I am ok with the single component reference from the yml file. I have started to do this anyway, as most of my components need to pass info between them and the easy way was to make a parent component that coordinates this.

2. I feel that there needs to be a deploy local option. Maybe the default could be to use the cloud method. I feel that if there is not, it will severely hinder adoption and I really want this to take off. I would think the biggest reason for this is security. We rarely give vendors keys and if we do, they are very tightly scoped. I don't think this would be possible since the service would need permission to create IAM policies/roles. Also, do the credentials have to be in that env.js? If so, this is a hard no-no for us. We don't allow the referencing of unencrypted secrets from the app/repo directory, even if they are gitignored. I recommend having an option to run the deployment service in your own account and it can send up metrics to Serverless, manage state, and do whatever else it needs to do. This is what a lot of serverless monitoring vendors do. Also, state management isn't really an issue for us. Every developer has their own stage that they deploy to and then our CI/CD service manages deployment to shared environments. Issues do happen especially because we are actively developing on the base components. This forces us to make our solutions very ephemeral and can easily be completely torn down and rebuilt, which really isn't an issue with the speed of components.

3. My other worry is that in my current development cycle I am making a lot of edits to the base components that my main app references, I see this as being very difficult in a non-local setting. I don't really see what the issue with NPM is. My parent components reference their child components locally inside the node_modules folder. I then handle where it comes from inside the package.json file. From here I can point the reference to either a versioned NPM package, a forked version inside my GitHub repo, or a local version that I am actively working on and let NPM handle the symlinking for me. How would an approach like this be done with this new registry?

Thanks for the work that you are doing. If you have questions please feel free to reach out.

[xiaoxinghu]

Where can I find the UI to search for components?