[docs] - Pipes API experiment

[erinkcochran87]

Summary & Motivation

This PR consolidates the Pipes and dagster-pipes API references into a single page. Prior to this, there were two references: one in API references > Core > Pipes and another in API references > Libraries > dagster-pipes. This fracturing of content creates a less-than-ideal experience for users, who have to look in two places for information about a single topic.

It could also be confusing, as the references weren't well differentiated. Ex: Do I need Pipes or dagster-pipes to do ?

The page for dagster-pipes remains, but it now contains only a brief description and a link to the relevant section in the consolidated API reference.

After:

Preview links: Dagster Pipes reference (consolidated); dagster-pipes reference

Before:

How I Tested These Changes

👀 , local

[github-actions[bot]]

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-2t3wjignt-elementl.vercel.app https://erin-pipes-api-experiment.dagster.dagster-docs.io

Direct link to changed pages:

• https://dagster-docs-2t3wjignt-elementl.vercel.app https://erin-pipes-api-experiment.dagster.dagster-docs.io/concepts/automation/schedules/testing

[schrockn]

Important to note that:

1. Writing and using pipes clients
2. Writing code in the external process that just streams metadata back to Dagster.

They are two different use cases targeted to different personas. The "platform owner" more likely to be working on and setting up clients; the "pipeline builder" will be including dagster-pipes in the external environment. Critical point: when someone includes dagster-pipes it does not include the dagster library.

So while the previous organization wasn't ideal merging all the pipes client stuff (which is much more complex) with dagster-pipes API reference also not ideal.

[schrockn]

Also heads up that I'm fixing pipes docs issues: https://github.com/dagster-io/dagster/pull/21719

[erinkcochran87]

@schrockn Heard. Took another swing at this - basically left all the formatting, but moved things back to their separate pages and beefed up the intros. I called out exactly where the APIs in the reference should be used and added callouts for users who may be looking for the other reference.

For example, I added this to the page for dagster-pipes:

... Looking to set up a Pipes client in Dagster? Refer to the Dagster Pipes API reference.

Note: This library isn't included with dagster and must be installed separately.

Updated dagster-pipes reference:

Updated Dagster Pipes reference:

[erinkcochran87]

@schrockn Made the changes you asked for! Let me know if you have specific wording in mind and I'll make the updates.