clay

clay copied to clipboard

We routinely face multi-day delays trying to integrate Clay into the liferay-portal repo due to various causes (but usually one of CI instability or fast moving upstreams which produce merge conflicts).

In addition, we get routine questions from teams about when a given feature will be available in DXP. For something simple like an icon update, it can take many weeks in the worst case; again, this may be due to multiple causes — like latency as the icon moves from a LEXI ticket to a GitHub issue through a PR to the Clay repo, via a release, and finally into liferay-portal after getting through review at the liferay-frontend fork, passing CI, and getting forwarded to and merged by Brian. This can be compounded by personnel bottlenecks, where not everybody has the knowledge or the comfort to take a change through all of the preceding steps. In theory, we do releases every two weeks, but because there are many steps and many potential sources of delay, things can get dropped on the floor or arrive later than one would hope given that some of the changes in question are so minimal.

Finally, there's the question of visibility, where even if things were to arrive at a regular cadence in liferay-portal, people outside of our team (and even sometimes inside of it) don't know when the next release is going to arrive or what is included in it (this was somewhat improved by #3917, which at least notifies subscribers to GitHub issues with a link to an LPS that they can watch in order to know when a release has arrived in liferay-portal). We have a general problem with traceability (as tickets move from LEXI to GitHub issues to LPS, and changes themselves move from fork to fork). Code itself arrives in liferay-portal via an npm install, so one cannot directly tell what is in the repo at any given time, or what is included in an update — the only visible trace is in a set of version numbers (example). ~If one wants to inspect the changelog, there may be as many as 40 separate files to inspect because each package has its own and we have a lot of packages~ (Global changelog was added in #3898). And when you do get in to a changelog, you may find that the "update" is basically empty like this one (a Lerna-generated thing that says "Version bump only"; there are some suggestions for ameliorating this in #3939). For an engineer, figuring out what particular Clay code is in effect in liferay-portal really means looking up a version number, downloading an npm module, and looking at the built artifact; note that you can't actually look up packages by tags in the GitHub repo any more because we don't currently publish per-package tags (ie. given a version number and a package name, you cannot jump directly to the corresponding code in the repo; instead, you have to carefully select the commit corresponding to the release date and the corresponding chore(release) commit).

So, labeling this as a thought experiment because obviously it would be a big change that would simultaneously bring great benefit and considerable cost, I'm making this issue to open up discussion of the question of "what if Clay where part of DXP itself?".

The obvious motivation here would be to solve all of the problems above:

• No explicit release process any more, so no questions about when a bugfix, icon or feature "arrives" in liferay-portal; as soon as it's merged, it's there.
• Better alignment with the way product teams and support works (where an LPS can be very directly tied to changes in liferay-portal).

The obvious cost would be that we would either lose or have to replace many of the supporting things we've built in this repo:

• GitHub actions would need to be reframed/reworked to operate in the context of liferay-portal CI.
• We'd need to reevaluate the role of clayui.com (would it live on in a separate repo like this one? if we dramatically change the release/development process, does it even make sense to maintain an independent marketing site like that any more?)
• We'd need to analyze and think about how to meet non-DXP use cases (and even the non-master branch DXP use case) and how we'd want to support those moving forward.

And probably a bunch of other difficult and thorny trade-offs. We would probably have to let go of some things that we'd rather keep a hold of in order to make this happen (because obviously, having this project in a separate repo affords us with a great deal of independence), but I think it's worth exploring the possibility because the cost that we pay to develop, maintain, and integrate Clay is very high and has been for a long time now, and it's quite possible that we could come out in front if we made some dramatic changes. (And even if we conclude that we don't want to or can't make such changes, I'm hopeful that the discussion can produce some ideas to improve our return on investment in this project by other means.

And just cross-linking the last existential discussion we had on a related topic, "What is Clay in relation to DXP?".

I was thinking the same thing last week for the AMD loader. In general, any project that it is used only inside liferay-portal should probably be put there because the development cycle is way much faster. Especially now that our build tools allow to use them in a frontend-friendly manner.

Regarding releasing for the outer public, we have sometimes assumed that embedding something in liferay-portal would make it "private", but that doesn't have to be necessarily true because we can have the projects inside liferay-portal and release to npmjs.com too, like we do with other artifacts (themes, JARs, WARs, etc.).

So, if we decide to do this for clay, I would like then to revisit the pros and cons for other projects as well (thinking now of npm-scripts, for example) once we accomplish this.

Very interesting discussion; I've often thought about the same thing.

Concernig the cost of doing this, here are some quick thoughts

GitHub actions would need to be reframed/reworked to operate in the context of liferay-portal CI.

Not sure how easy or hard that would be but totally worth it according to me

We'd need to reevaluate the role of clayui.com (would it live on in a separate repo like this one? if we dramatically change the release/development process, does it even make sense to maintain an independent marketing site like that any more?)

I'd say that it probably should, once again, just my 2 cents

We'd need to analyze and think about how to meet non-DXP use cases (and even the non-master branch DXP use case) and how we'd want to support those moving forward.

Yeah that's a very good point and I'm also interested in participating to this discussion.

I don't have much more to add than what you and @izaera already said, but I'm very interested in moving this forward whether it's participating in the migration or testing whatever needs to be tested.

Greg's intro looks really sound. I feel that answering "why Clay needs to be in a separate repo?" might provide a counter-argument to be compared against that intro, and re-evaluate with today's eyes. That exercise may be done, as @izaera suggests, with any other product/tool/library we have in separate repo and is meant to be part of the DXP bundle.

I would kindly let the most veteran ppl in the team to provide the needed context to understand why those started separately, so that we have a good source of trust.

Finally, my support background leads me to carefully consider the non-master side of things.

My main concern is how this would impact the way Clay is developed?

Would Clay still have an instant feedback loop while developing new components or maintaining existing ones? Or would it require running DXP/Gradle/whatever makes our feedback loop slow on DXP.

After spending several months working on Clay I dream of a day I can have hot reloading and other mechanisms that make development fast over there. Taking that away from Clay would suck.

From a Support point of view, I'm more than happy to see this proposal. I think this would help us in many ways as Greg pointed out:

• To have a similar procedure with any regular fix in portal
• To make easier figure out what changes have been included in every fixpack (right now, if you have a look in our Customers Help Center, many changes can be masked under 'Update Clay to v.X.XX.X')
• Speed up the procedure of backport and deliver solutions to customers

As usual my concerns are related to maintenance branches. Right now we have Clay as an npm module in 7.0, 7.1, 7.2 and 7.3. Since this would be a long-term task, I'm assuming it wouldn't be backported to any of those branches, so even after integrating Clay in Portal (let's be optimistic and say in 7.4), we'd need to maintain the npm module until 2027 (end of 7.3 service Life). Obviously it is not going to require a ton of work since we are not fixing bugs every single day, but there is a possibility to need a fix and a new release until that date.

My other concern is about customer's developments and especially how to upgrade them. I mean, if we need to keep maintaining npm modules because of what I said above, any customer with a custom development in 7.2 is going to be glad importing Clay from npm. But what if this customer upgrades to 7.4? We need to keep that in mind and try to provide them with the tools to make the upgrade of their own code as smooth as possible.

TLDR; I'm not fully opposed, but I also think we might be trying to fix our problems with the wrong solution. I would vote to keep them separate and focus on the specific problems at hand.

It's tempting to think that by merging into liferay-portal, we fix all these problems because now the problems we face with an external repo are just the same ones we face with everything else in DXP. But I think this really boils down to "What are the problems and whats the best way to fix them?".

From the comments above, it seems like the primary issues are

1. Inconsistent release cycle
2. Lack of visibility and traceability
3. Long delays when integrating Clay into DXP.

Is merging Clay into DXP the best way to solve these issues? Personally, I don't think so.

Instead of listing the problems with Clay being in DXP (I think there are a handful), let me see if I can explain why I think those three primary issues can be fixed with our existing structure.

Inconsistent release cycle Before 7.3, we had a fairly regular release cycle. Every 2 weeks, we would cut a release on Tuesday and send a PR to liferay-portal. As we got closure to 7.3, things became a bit more wild and we started to cut releases multiple times a week and didn't lock into a specific milestone. Since that happened, we have been much more inconsistent with releases and the entire process. I think if we establish something more rigid, we would see more consistency. Additionally, I also think we are inconsistent with this because in some sense we have said that Clay is a "whole team responsibility", which I think leads to a Diffusion of Responsibility. I think if we were to say it is {x-person}'s responsibility this week/month/forever, there would be more consistency in releases.

One note on releases, it's often hard to know the priority when it comes to clay issues and when they should be worked on and if they are necessary for release. So we might want to think through how we prioritize clay issues vs other issues in our Episodes. I think this is a side-conversation though

Lack of visibility and traceability I think this is more of a tooling and process related issue than it is a code location issue. We never really incorporated or set out a formal process for how we maintain visibility of changes for when we merge Clay into portal. I've seen comments or questions pop up here about what is in a certain version or if the version is released or merged into DXP yet, but we never formalized how we actually want to communicate that during the merge/release process. I think if we set out a task to formalize this and improve tooling, we could dramatically increase our visibility and traceability. For example, if we could generate a liferay-portal focused change log that gets added to each release ticket(LPS) and PR. That would make sure support and others wouldn't have to track things via npm releases or our package change logs.

Long delays when integrating Clay into DXP. This issue seems to be the one of the biggest headaches. Seeing an "Update Clay" PR fail CI after 6 hours of CI can making any person want to cry. To be honest, I don't really know what we do to fix this but I don't really think this is an issue that stems from Clay being in a separate repo. Any time we update 10s of modules across DXP, CI seems to always be a nightmare. This is not a Clay issue, this is a "changes across multiple modules" issue. This was an issue when removing metal as well. We might benefit from exploring a more automated and continuous testing of latest Clay in DXP with CI, that way we can catch issues before running 6hr CI for a needed release.

---

In regards to "why does Clay does clay need to be in a separate repo?"

• Fast development cycle in isolated environment (the biggest reason IMO)
• Isolated tests and CI
• Multiple package management
• Docs co-located with source code
• Usable outside of DXP
  • Was more important when AC and Commerce were outside of liferay-portal

Anyways, thats my long list of thoughts. The end

Would Clay still have an instant feedback loop while developing new components or maintaining existing ones? Or would it require running DXP/Gradle/whatever makes our feedback loop slow on DXP.

After spending several months working on Clay I dream of a day I can have hot reloading and other mechanisms that make development fast over there. Taking that away from Clay would suck.

The theory is that once we integrate webpack 5 we can have hot reload inside liferay-portal (don't know yet if for the full portal, but definitely doable for one module at a time). That would make it very fast, I think, because you would be able to modify clay inside frontend-taglib-clay and instantly see the changes in the whole portal.

That said, until we see it running in Real Life :tm:, I wouldn't rely on it for making this decision.

Another reason why we might want to consider merging to DXP is so that we can update markup in both components and tags at the same time. Or maybe there is another way we can avoid having to update markup at two different times.

I'm curious to hear from more people on this though.

Another reason why we might want to consider merging to DXP is so that we can update markup in both components and tags at the same time.

I'm in. @carloslancha @markocikos @marcoscv @edalgrin @pat270 probably won't mind either.

One additional benefit I see is that it would be easier to get contributions on issues like ClayPanel: The page is refreshed after click on the Header, inside a Form Tag.

I showed this debate to some Product Team Leads who saw it also as a potentially positive move that would also benefit other devs in terms of showing good practices, advancing frontend build processes for everyone and simplifying dependency and version management.

I think @bryceosterhaus's list of concerns should not be quickly dismissed though:

In regards to "why does Clay does clay need to be in a separate repo?"

• Fast development cycle in isolated environment (the biggest reason IMO)
• Isolated tests and CI
• Multiple package management
• Docs co-located with source code
• Usable outside of DXP
  • Was more important when AC and Commerce were outside of liferay-portal

I don't think this necessarily needs to stay a separate repo for those reasons but rather that we should work towards bringing those benefits in some form to liferay-portal if we want to consider it

[...] we should work towards bringing those benefits in some form to liferay-portal if we want to consider it

This is the biggest thing IMO, removing the technical debt as we are is probably a good start, but overall getting a similar development cycle into DXP would be very much ↓

you guys write a lot! 😄 apart from an uneasy feeling of losing control over Clay and start injecting things we don't want in it, I don't have any real concern about this

One additional benefit I see is that it would be easier to get contributions on issues like ClayPanel: The page is refreshed after click on the Header, inside a Form Tag.

I hadn't thought about that before, which I think is another good reason. Although, technically people can still contribute back, it just requires more effort. Or maybe we provide a doc on how to contribute to Clay coming from a DXP-centric perspective. Show devs how to make a change in the clay repo and test it in dxp easily.

Re-reading the list of issues I had brought up earlier, one particularly stood out and that was the area of documentation. I don't know we would be to support our current clayui docs or storybook demos. If we did move to Clay to portal, I think we would have to re-evaluate having "demo-able" docs since we wouldn't be publishing to npm. Which also makes me realize that this would also limit our ability to use clay components in codesandbox for examples and bug reproduction.

So with that in mind, I think I would still be pretty far on the "keep clay separate" side.

Well, it took me a while to leave my opinions about it but I wanted to see what everyone thinks before commenting on something but I have some concerns in mind, some are the same that were commented on in this thread.

Looking more closely at the conversations a change that I realize is that being in DXP it looks like we would change Clay’s focus a little bit, right? or it's just my impression! Clay may not be Clay anymore as we think ... he will probably adopt more components or specific use cases for DXP. My concern about this is that Clay's development was not aligned to specific cases but it was developed thinking about trying to be more generic with Lexicon in mind whenever possible so that different teams can adopt, not considering only DXP, I believe that this is the future for us since the company is running after building other products or leveraging what we have to increase revenues, this allows teams like AC and maybe Liferay Cloud to use it. I initially had some experiences with them using Clay on the Admin Console, it was really cool and the feedback was very positive as it allowed them to develop fast...

The part I like best about keeping Clay separate is that we have a lot of control over what goes in and create our own processes:

• Preview Deployments, Storybook and Clayui.com (This helped a lot during PR reviews)
• [1] Our CI is considerably much faster than DXP, we can send things much faster.
• We have a stable and development branch, this allows us to send things in progress to the master without worrying about breaking anyone. (I'm not going to say that we do that a lot until after the launch we focus more on a stabilization phase and not creating new things and we focus on other things)
• Keeping the documentation together with the code was a big benefit for us and we managed to keep the site more up to date as we changed things. For those who were not here before, the documentation was separate from the components as well as the CSS and we had a lot of traceability problems with what was changed and our documentation was always out of date.
• Documentation is something important as a website, it can be seen as marketing but I think more about creating relationships, this seems to have been proven in some way.

With that in mind and putting it in DXP I see that we can lose some things and maybe win some but I am still not 100% convinced of that.

• The portal development cycle is painfully tiring. (I think I’m not very lucky with that 😭 but in my experiences, on the Portal, they are always problematic, mainly due to the CI and the sync time from Brian’s branch to the master)
• I have the concern that when the code is inside the Portal it can become distorted since anyone can change and sometimes it happens that someone changes without sending it for review.
• DXP follows only one branch practically, what I like to tell people when I am talking is that DXP has the master branch as a production environment, where you cannot develop quickly and have a long CI as well as the final stage of a deployment cycle for production.
• We probably won't be able to version control packages within DXP, everything that enters DXP is in the master version (Even if we can release to npm from within the portal it can be a little more difficult).

One point to consider that I saw a lot being commented on Slack if I am not mistaken is that Clay release PRs always take time to get in, it can be a little unfair to talk about because it is the "normal" DXP process and even if everything is in of DXP we will still have problems like this but now in several PRs we send, where a release that has many corrections passes once in the CI we will have this multiplied by the amount of fixes. Of course, this may not be very real, but we must take it into account.

Release cycle

This probably seems to me to be one of the biggest pains for the reason to move to DXP, it is probably when an issue reported by a team takes time to enter DXP. There are probably improvements on our side without having to change everything in a very drastic way.

• Our launch cycle today is 2 weeks, we can experience shorter release cycles.
• We can work in the process of canary releases, as mentioned in other issues and Slack
• We can have shorter releases for issues blockers or just those that are reported by the team
• The icons seem to be more painful, once we add them we can always create the release of the icons and send them to DXP

Collaboration

Well, I don’t know if this is a problem on Clay’s side being a separate repo, in fact, the teams don’t have that culture, the culture is more the product silo, you care about your product and something that prevents them, the demand is now from the other team. That's what I see at least in Brazil!

The collaboration that exists speaking more in Brazil is that when they think there may be a problem with Clay components or anything from the Portal they call us them to identify the problem and once it is done they open an issue or ticket and will focus on other tickets.

---

I am not 100% opposed to these changes but switching to DXP I am not 100% convinced that the exchange is more beneficial than the loss of certain resources. I think we can better tackle these bottlenecks so that it doesn't require a drastic change. Finally, I am sad that certain things, in the end, end up returning to keep everything within the DXP monorepo and we continue to grow this big repo over time but without changing the merging process.

[1] It does not mean that our tests are more important than that of poshi, but that I consider this last stage as "launched for production".

Thanks everyone for this insightful discussion. As as designer, there's a lot of technical details that I'm not able to fully understand, but I have a general sense of the pain points stated. I appreciate what @bryceosterhaus stated about considering alternative solutions for the problems identified.

The value of docs My experience with Clay has been with working on the Workflow product team as well as client engagements for Global Services. A few thoughts I'd like to share from a designer perspective are that the clayui.com docs and Storybook are immensely valuable in knowing the capabilities of components from the API to the interactive playground. These resources help designers understand and therefore decide how to create the best user experience for the feature at hand as well as closing the gap with developers on how to communicate interaction and functionality. Not to say that these resources themselves must remain as is, but the value they provide is truly irreplaceable so if clay goes into DXP, then I hope there will be a resolution to this need. Thank you so much for your diligence to update and optimize the docs!

Clay for GS My other main concern from the Global Services perspective is that assuming clay is within DXP, when a client chooses to build on a specific version let's say 7.3 or 7.4, would they be bound to clay's functionality at that release or could it still be a separate package that can be updated separately? I assume that it would not be able to be updated and must be 'used' at the functionality in the DXP release version. If I'm not mistaken, then designers will need to design within that constraint of clay at that DXP release. This isn't a dealbreaker by any means as we will face all kinds of constraints on a project, but it's something to note as a limitation when we are ideating a solution for the client's requirements. If this were all true, then ideally we would want or even need to have documentation, examples and a Storybook like playground for each DXP/Clay version so that what is designed can be implemented within that client's version of DXP/Clay. I hope I'm making sense haha.

If anything I just wanted to communicate that GS engagements thus far have been heavily dependent upon using clay to build the UX/UI for all kinds of solutions and obviously with some additional custom components or extending clay ones. Moving forward I don't know if this will always be the case as DXP and other Liferay product needs are evolving to be more complex and specific while the components for client's requirements perhaps may be better implemented with a different library at that point. I think GS has always built with clay for many beneficial reasons and it has stayed flexible/universal for not just serving DXP/Liferay products' needs. I'm just wondering if this will still be the case looking into the future of Lexicon + Clay.

Lastly, I intentionally didn't speak about Lexicon as a designer as I would leave that to the Lexicon designers to articulate their concerns and workflow from the trenches. I do always refer to both Lexicon and Clay docs to ensure that I am designing according to consistent patterns though for both DXP product design + GS projects. I hope sharing my experience was helpful! 😅

A lot of the replies on this thread so far have consisted of "but ... we'll lose [something] if we do this!"

I tried to foreshadow this — albeit while staying away from prescribing solutions because I wanted the discussion to be as open as possible — in my original description when I said (emphasis added):

we would either lose or have to replace many of the supporting things we've built in this repo:

• GitHub actions would need to be reframed/reworked to operate in the context of liferay-portal CI.
• We'd need to reevaluate the role of clayui.com (would it live on in a separate repo like this one? if we dramatically change the release/development process, does it even make sense to maintain an independent marketing site like that any more?)
• We'd need to analyze and think about how to meet non-DXP use cases (and even the non-master branch DXP use case) and how we'd want to support those moving forward.

So just to be super clear: I am not proposing that we jettison the docs or the storybook examples. I never said that having the code living in liferay-portal would or should preclude us from cutting npm package releases from that code base. And I am still appalled at the historically huge tax we have paid for running Clay as an external project. It wasn't all bad (for example, it enabled us to move quickly towards a more modern stack involving TypeScript), but that historical benefit doesn't mean that we shouldn't re-evaluate things in light of what makes sense for the future. liferay-portal has evolved and continues to evolve too; we can consider and use technologies and workflows in liferay-portal in 2021 that really weren't on the table back in 2018 and prior. We should be optimizing for the common/main use case, but we seem to be doing something else. Note that optimizing for a use case does not mean excluding other use cases; it just means making the best use of company resources in the interests of the company.

One year later (time flies), I'm revisiting this issue and I think it's about time we took a decision.

Let's talk once again about the lists of concerns that were mentioned by @bryceosterhaus

In regards to "why does Clay does clay need to be in a separate repo?" Fast development cycle in isolated environment (the biggest reason IMO) Isolated tests and CI Multiple package management Docs co-located with source code Usable outside of DXP Was more important when AC and Commerce were outside of liferay-portal

I just wanted to share what I think about all these points to see if we're able to take a decision

1. Fast development cycle in isolated environment

What we're referring to here has a name and it is called "storybook" (correct me if I'm wrong), so as long as we can make that work in DXP (or to be more precise, in the OSGi module where Clay would be located) it shouldn't really change.

2. Isolated tests and CI

We can run isolated tests, just like we do in other OSGi modules but as far as the CI is concerned that's not really an option (and the CI doesn't seem to be more stable today than it was when this thread started). We could explore the possibility of executing our GitHub Actions workflow locally or even on our CI but we'd need to collaborate with the CI team for that to happen. It might be a good idea to see if all of those GitHub Actions are still necessary, just to make sure we don't bring stuff that's not needed.

3. Multiple package management

That would require some work, but since we're using yarn workspaces in DXP, it might mean that this is actually possible without too much effort, at least I think it's worth trying at the risk of seeing my terminal spill out lots of error messages (wouldn't be the first time).

4. Docs co-located with source code

That's the only point I'm not sure about. I'm not saying we're doing a bad job at this, but we know from the feedback we received that our documentation isn't good enough: or at least, our users don't think it is.

"Automating" this documentation is useful because it saves us time, but that doesn't mean it produces something people like. Maybe it's time we rethink this or maybe this needs to be more manual because people want more examples, written in a different style. In any case, we will have to work on that, whether we move Clay to DXP or not, and probably also include Lexicon in the process to make sure we're on the same track.

5. Usable outside of DXP

As long as we can publish packages to npm that should be possible. On another hand, and to be honest, I don't see Clay being used outside of DXP that much (looking at the issues, pull requests, etc...) and neither do I think it needs to be that way.

So I think we've got two options:

• Work on making points 1, 2 and 3 happen and talk about point 4 and 5 (and then work on those points)

• Decide that we'll never do this and close this issue with our final decision.

Let me know what you guys think when you can, I'll be creating a poll in Slack about this topic soon(ish).

What we're referring to here has a name and it is called "storybook" (correct me if I'm wrong), so as long as we can make that work in DXP (or to be more precise, in the OSGi module where Clay would be located) it shouldn't really change.

I think Storybook already works within the DXP module, but that's not just the point here about the development cycle, the CI/CD is the important part of this workflow, meaning we can send changes quickly to the repository, for example, we can change a line and in a few minutes merge, unlike the DXP we usually waste 1 day running the CI, I think it's more tied to that, for example, this flow also changes to @pat270, since it sends a lot of PRs from changes, this tends to congest the PRs longer to make the changes.

We can run isolated tests, just like we do in other OSGi modules but as far as the CI is concerned that's not really an option (and the CI doesn't seem to be more stable today than it was when this thread started). We could explore the possibility of executing our GitHub Actions workflow locally or even on our CI but we'd need to collaborate with the CI team for that to happen. It might be a good idea to see if all of those GitHub Actions are still necessary, just to make sure we don't bring stuff that's not needed.

Well maybe this works we can talk about it but I think we'll still have to run ci:test:relevant with every new PR so that's one more thing, the good thing we just run the DXP CI when we've had more than 20 PRs merged into Clay to release.

The only point here is that we have releases every 2 weeks, so things take a while to get in, but we usually have an agreement that if it's a release blocker we don't see problems fixing it, making the release and sending it to DXP. Since on our side the release is fast but CI is our blocker to get things moving fast in this use case.

That would require some work, but since we're using yarn workspaces in DXP, it might mean that this is actually possible without too much effort, at least I think it's worth trying at the risk of seeing my terminal spill out lots of error messages (wouldn't be the first time).

Yeah, maybe it works, yarn workspace is more tied to package.json, so I don't know if an OSGI module could have multiple folders with package.json. Although now we only want the core package maybe we don't need everything else we have clay-css, it could be in another module maybe.

It's also very related to the release cycle in NPM, using lerna, for now we're still keeping the packages separate maybe until v4, where we can stop supporting this to just use core.

"Automating" this documentation is useful because it saves us time, but that doesn't mean it produces something people like. Maybe it's time we rethink this or maybe this needs to be more manual because people want more examples, written in a different style. In any case, we will have to work on that, whether we move Clay to DXP or not [...]

Yeah, regardless I think we should improve our documentation, I've been talking about this for a while, so we just need to prioritize it. Some issues were created for us to start working on organizing the documentation as well.

On another hand, and to be honest, I don't see Clay being used outside of DXP that much (looking at the issues, pull requests, etc...) and neither do I think it needs to be that way.

It's quite tricky to metrics if Clay is used outside of DXP, for example, the Liferay Slack community sometimes seems to use Clay and for some questions, I answered they are pretty active waiting until components like TreeView get ready to decide if the team is going to use or build one of their own, another example of DXP out of use is Liferay Cloud which has an admin dashboard built entirely using Clay, Analytics Cloud has also used less.

I've seen the Solutions team also use Clay to remote apps for some of their projects. So it's really hard to say that Clay is not used outside because we don't have a way to metric this, we also need to know which direction Clay and Lexicon want to take, do we want our clients to use Clay, or do we just going to develop this for DXP? cc @dsanz.

Over the time I'm at Liferay and Clay, I see different views on this and it's changing all the time in various discussions, I always see someone saying that we should just worry about DXP and then someone saying that we should worry about customers using 🤷‍♂️. So if Lexicon and Clay agree that both should only be used internally, I really don't see any reason to keep this open anymore.

Before making a decision like this that affects many people, we must make a plan before starting to move everything.

• What will the backport cycles be like for DXP versions? normally this is done by just changing Clay's version very easily.
• How we will update the site documentation and Storybook work, even though we don't have the best documentation and examples, both are live, and this is used by everyone. Before we had a lot of inconsistencies, I remember to move Clay's components to be closer to the CSS and the website so that the workflow was closer, that is, building the documentation and examples and seeing them being changed in the same repo, even if we have the storybook in DXP, it cannot be published, neither can the documentation. So we're back in the past of keeping them separate and we need NPM packages being published to add to the storybook and clayui.com if it's in a separate repository.
• Probably we will have an impact on our productivity due to the CI DXP and, more continuous ant all, also the reviews will take longer because we will have to ant all to test the changes, normally I just open the link generated by Netlify.
• We will have to keep the packages being published so we still need the 43+ packages in DXP with some support from lerna or something to help with the release in NPM.
• We probably won't have more releases every two weeks, I don't know if it makes more sense in that case, maybe just for NPM, but we also need to put the changelog somewhere. We currently have CHANGELOG.md and support from the github repository releases page.
• We have a lot of things organized in our repository and internal processes like CI, contribution guides, internal documentation... this will probably be lost, did we get something like that in DXP? this would still be visible or just one file lost to more than a thousand files in monorepo.

Just gonna chime in on the CI portion. I think CI is my biggest concern and perhaps the most difficult of these challenges to overcome since it effects time, people, and infra resources. The two biggest issues in CI and PR workflow I see are

• Running ci:test:relevant on every clay PR causes long delays
  • Increases work for QA analysis with flakey PRs
  • Increases load on CI infra
• Lose ability to have deploy previews w/ netlify and storybook

One benefit with CI is that we have continuous results with Clay rather than the bi-weekly battle with CI trying to get a clay release merged. All of the other notes that Matu brought up are really valid too though.

Lastly, since we could probably talk about potential issues for a long time, we may want to consider just trying it for a few weeks and see how it goes, that seems like it might be the best way to evaluate if our concerns are as big of an issue as we think. Trial and error here seems like the best potential way forward. It doesn't have to be perfect, but just attempting it might give us a better idea of how beneficial or costly it really is. ^ the cost here is potentially a few weeks of lost time

@matuzalemsteles and @bryceosterhaus thanks for the additional information you both brought up interesting points. We have our bi-weekly Clay squad meeting today (at 17:30pm GMT+1) in which we'll talk about this - and update or close this issue, once we decide what we want to do about this.

Closing this issue, this is a long discussion on this and we changed principles of how Clay should be offered to DXP or to other customers during these years, several other initiatives have emerged how to provide Clay to customers and continue to be a separate package of DXP as an incentive for Client Extensions that some teams have been using to develop solutions for clients. Also in the last few years we evolved Clay and refined how we provide components better handling accessibility and OOTB features that developers don't need to implement.

We also did some research on moving Clay to DXP a while back https://liferay.atlassian.net/wiki/spaces/ENGFRONTENDINFRA/pages/1972797505/How+we+plan+to+make+Clay+easier+to+use+and+extend+for+DXP+developers.