Paste API

[dmos62]

Fixes #2844

Implements paste API.

Initially we envisioned a generic bulk upsert API that could be used by the frontend paste functionality, hence the title of #2844. I explored that idea extensively, but opted for a non-generic API specifically for pasting.

In my estimate, the bulk upsert as expected by a user using paste in a spreadsheet is wildly different than the bulk upsert expected by database administrator doing INSERT ON CONFLICT DO UPDATE. Trying to accomodate both seemed like a good way to waste a lot of time and end up with bad UX for both.

Instead of summarizing my concerns, I'll summarize the features of the proposed paste API:

updates = [
	dict(
		changes=dict(
			Status='0',
		),
		conditionals=dict(
			id=record_0_id,
		),
	),
	dict(
		changes=dict(
			Status='1',
			Center='2'
		),
		conditionals=dict(
			id=record_1_id,
		),
	),
]
inserts = [
	dict(
		Status='3',
		Center='4',
	),
	dict(
		Status='5',
		Center='6',
	),
]
data = dict(
	updates=updates,
	inserts=inserts,
)
client.post(f'/api/ui/v0/tables/{table.id}/records/paste/', data=data)

Above is the basic outline of the API. The endpoint accepts (via POST) a dict, that has two fields: updates and inserts.

It's assumed that frontend will always know whether a given row being pasted should result in an update or an insert. Making that explicit prevents surprises in case of relevant changes having been made that were not synced with the calling frontend.

Updates are a sequence of dicts: each dict has a changes key and a conditionals key: changes define the field-value pairs to change in the row that is satisfied by the equality comparisons defined by field-value pairs in conditionals. Think how UPDATE works in SQL. An update with changes being {'field1':'val1'} and conditionals being {'id':'val2'} requests that for the single row whose id field equals val2, its field1 field's value is set to val1.

If more than one row would be updated by a given update, that's a fatal exception, because when pasting a row it should never result in more than one updated row. The paste API attempts to minimize surprises to the end user.

Inserts are a sequence of dicts: each dict defines the field-value pairs that should be set on the new resulting row.

An insert will always result in a new row or will panic if that's impossible for whatever reason.

Any panic encountered will rollback all changes made. Paste is meant to be an all-or-nothing operation.

Also, this API panics if any user-defined changes to SERIAL or IDENTITY columns are requested. These column types need special handling when mutating their values, because there are sequences involved, and that seems hard to safely generalize (i.e. prevent surprises).

All updates are performed before any inserts. All updates are performed in order requested; same for inserts.

Checklist

• [x] My pull request has a descriptive title (not a vague title like Update index.md).
• [x] My pull request targets the develop branch of the repository
• [x] My commit messages follow best practices.
• [x] My code follows the established code style of the repository.
• [ ] I added tests for the changes I made (if applicable).
• [ ] I added or updated documentation (if applicable).
• [ ] I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[seancolsen]

@dmos62 I assume this PR is to fix #2844, is that right? If so, can you update the PR description?

[seancolsen]

@dmos62 can you give an update one where you're at with this PR? How close to completion is it? What work is remaining?

[dmos62]

I'll update the top post with what the goal/spec of the PR is.

As to the progress, here's the summary, but keep in mind that I might not recall some details. The PR seems 80% complete. The SQL logic and tests seem to be there. The API tests are there for the update aspect of pasting. The API tests for the insert aspect of pasting are missing. The API tests for the paste operation being all-or-nothing are missing as well.

[mathemancer]

Closed for now, but we may grab some pieces of this later.