Standard library brainstorming

[axic]

This was recently mentioned in the EthOnline Future of Solidity talk, but we have been discussing this here and there for the past months, and goes back as early as #228.

The goal is to move functionality out of the compiler into code written in Solidity and slowly build out a standard library -- lets call it stdlib. Files/contracts/functions from the stdlib would need to be explicitly imported, and would not be exposed implicitly. Initially this file would still be part of the compiler source code and included as a literal in the binary, but potentially in the future it could live outside should importing from content addressible locations turn more naturally supported by tools.

Functionality we want to move out / include:

1. Function available in the global namespace
2. Functions attached to certain types (such as those on address)
3. Potentially things like block, tx, and msg
4. Helpers to be used within assembly

There is some initial work available in https://github.com/ethereum/solidity/tree/stdlib

[chriseth]

I think an option to export the stdlib to the filesystem could be a nice addition to the commandline: solc --export-stdlib /my/folder/to/store/it. Then users could choose to use the built-in stdlib or the ones in a local folder. We have to decide what to do on a name clash. Maybe it would be fine to allow both to be imported in the same way and check for equality. If there is a change, it has to be signalled somehow in the metadata.

[chriseth]

Should we create a base class "TokenOwner" that has functions to transfer tokens? Is this a standard somewhere?

[axic]

Should we create a base class "TokenOwner" that has functions to transfer tokens?

What is that exactly?

However we said not to include ERC standards in this library, at least not at the start, but rather focus on moving parts of the compiler out.

[smartcontracts]

A wider range of type casting options would be wonderful.

[leonardoalt]

A wider range of type casting options would be wonderful.

@kfichter Lately we've actually been making type casting stricter and more explicit. What would you suggest could be improved?

[smartcontracts]

@leonardoalt honestly the more I think about this, the more it seems like the sort of type casting I'm interested in is probably beyond the requirements of a stdlib. Probably better for this sort of stuff to live in a different (but standardized) library.

[smartcontracts]

Although other common operations like bytes manipulation (slicing, for one) or bit-level stuff (checking if a bit is set) would be very useful. It's easy enough in assembly, but would be nice not to have to implement it ourselves.

[axic]

In https://github.com/ethereum/solidity/issues/9358#issuecomment-655626101 there was as suggestion for other special namespaces like crypto.

[cameel]

One thing that might be useful in stdlib is the equivalent of bytes.concat() for strings (unless we want to add string.concat()).

I've seen an OZ issue about string utils where the topic of concatenating strings came up and something like this was suggested:

string(abi.encodePacked(partOne, Strings.toString(numberTwo)))

with bytes.concat() it would be:

string(bytes.concat(bytes(partOne), bytes(Strings.toString(numberTwo))))

but that's even longer than the original. Would be nice to have

string_concat(partOne, Strings.toString(numberTwo))

[chriseth]

It's still not totally clear to me how the import mechanism works. I created a hackmd note to discuss: https://notes.ethereum.org/@solidity/r1lNFFdjec

[axic]

Since this has been a bit stalled, perhaps the best way would be adding a new experimental flag to the compiler (either pragma experimental stdlib; or a compiler flag) which turns off all the current builtins and supports the special import statement.

This way we could merge functions step-by-step even before we get generics supported.

[axic]

Based on the internal architecture of GlobalContext it may be less intrusive to do this via a compiler flag, but that would mean external libraries can't really be used in the experimental stdlib pipeline.

[axic]

It's still not totally clear to me how the import mechanism works. I created a hackmd note to discuss: https://notes.ethereum.org/@solidity/r1lNFFdjec

Copying the (I think) "decisions" from there:

1. Consider the std path as the stdlib, e.g. "std/operators.sol" would import from there. Raise an error if some remapping/path is in conflict.
2. solc --export-standard-library <target directory> (and its standard json equivalent) would dump the internal stdlib sources.

[axic]

Experimented with this a idea a bit, pushed the work here: https://github.com/ethereum/solidity/tree/stdlib2

I think this is workable, we can slowly introduce parts of the stdlib without waiting for all the needed language features to be present. This will mostly be enough to allow globals, and potentially some member functions (with using global 😬).

[axic]

Created https://github.com/ethereum/solidity/projects/46 to group the required/considered features.

[axic]

Moved some of the example code we had lying around to https://notes.ethereum.org/@solidity/Sy03Y8_c5 (most of this is incomplete)

[cameel]

Decision from the call: we'd prefer a special syntax for the stdlib imports:

import std.cryprography;

[cameel]

Also, instead of pragma stdlib we'll roll it into the pragma experimental solidity-next planned for enabling the upcoming breaking changes in the language.

Just for the record, some things we considered on the call:

• It should not be too annoying to change the imports when swapping out the stdlib. So maybe it would be better to use / instead of . and to keep the extension in there: in the end we decided against both. We'll go with syntax I posted above.
• Having the same syntax for stdlib imports no matter whether the source uses built-in stdlib or one provided by the user: rejected. @ekpyron thinks it's better for swapping out the stdlib to be an explicit change in the source.
• A pragma for defining a path to user-provided stdlib, as an alternative to changing every import: not entirely rejected and we may do this in the end, but for now we decided to keep it simple and stick to having to adjust imports.