libtock: Rewrite and standardize API

[bradjc]

Rewrite libtock-c with a standardized format. See the guide.md file.

Progress

• [x] chips
  • [x] remove all chips
• [x] crypto
  • [x] aes
  • [x] hmac (@bradjc)
  • [x] sha (@bradjc)
• [x] display
  • [x] screen
  • [x] text_screen
• [ ] interface
  • [x] button
  • [x] buzzer
  • [x] console tricky (@alevy)
  • [x] led
  • [x] usb_keyboard_hid
• [ ] kernel
  • [ ] ipc tricky (@alevy)
  • [x] read only state (@bradjc)
• [ ] net
  • [ ] ble (@alevy)
  • [ ] ieee802154 (@tyler-potyondy)
  • [x] lora_phy (@bradjc)
  • [x] nrf51_serialization even worth doing??
  • [x] udp (@tyler-potyondy)
• [ ] peripherals
  • [ ] adc (@alevy)
  • [x] analog_comparator
  • [x] dac (@bradjc)
  • [x] gpio (@bradjc)
  • [x] gpio_async (@bradjc)
  • [ ] i2c_master (@alevy)
  • [ ] i2c_master_slave (@alevy)
  • [x] rng
  • [x] rtc
  • [ ] spi (@hudson-ayers)
  • [x] spi_peripheral (@alevy)
  • [x] usb (@bradjc)
• [x] sensors
  • [x] ambient_light
  • [x] humidity
  • [x] ninedof
  • [x] pressure
  • [x] proximity
  • [x] sound_pressure
  • [x] temperature
  • [x] touch
• [x] storage
  • [x] app_state
  • [x] kv
  • [x] sdcard
  • [x] nonvolatile_storage
• [ ] OTHER (not sure how to group)
  • [ ] alarm/timer tricky (@hudson-ayers)
  • [x] crc
  • [ ] unit_test

Apps

• [ ] music
• [ ] witenergy
• [ ] unit_tests/nrf52840dk-test
• [ ] touch
• [ ] tests/lsm303dlhc
• [ ] tests/ieee802154/radio_tx
• [ ] tests/ieee802154/radio_ack
• [ ] tests/ieee802154/radio_rxtx
• [ ] tests/ieee802154/radio_rx
• [ ] tests/hmac
• [ ] tests/udp/udp_virt_app_kernel
• [ ] tests/udp/udp_virt_rx_tests/app2
• [ ] tests/udp/udp_virt_rx_tests/app1
• [ ] tests/udp/udp_send
• [ ] tests/udp/udp_rx
• [ ] tests/udp/udp_virt_app_tests/app2
• [ ] tests/udp/udp_virt_app_tests/app1
• [ ] tests/touch
• [ ] tests/crc
• [ ] tests/i2c/i2c_master_ping_pong
• [ ] tests/i2c/i2c_master_slave_ping_pong
• [ ] tests/hail
• [ ] tests/spi/spi_byte
• [ ] tests/spi/spi_controller_transfer
• [ ] tests/spi/spi_peripheral
• [ ] tests/spi/spi_buf
• [ ] tests/spi/spi_peripheral_transfer
• [ ] tests/multi_alarm_test
• [ ] tests/analog_comparator
• [ ] tests/app_state
• [ ] tests/microbit_v2
• [ ] tests/imix
• [ ] tests/aes
• [ ] security_app
• [ ] ble-uart
• [ ] courses/2018-11-SenSys/sensys_udp_rx
• [ ] courses/2018-11-SenSys/important-client/app2
• [ ] courses/2018-11-SenSys/important-client/app1
• [ ] screen
• [ ] tutorials/03_ble_scan
• [ ] tutorials/05_ipc/led
• [ ] tutorials/05_ipc/rng
• [ ] tutorials/02_button_print
• [ ] tutorials/hotp/hotp_starter
• [ ] tutorials/hotp/hotp_oracle_complete
• [ ] tutorials/hotp/hotp_milestone_three
• [ ] tutorials/hotp/hotp_milestone_two
• [ ] tutorials/hotp/hotp_milestone_one
• [ ] ip_sense
• [ ] lora/sensor-transmit
• [ ] lora/sensor-receive
• [ ] find_north
• [ ] rtc_app
• [ ] services/ble-env-sense
• [ ] services/ble-env-sense/test-with-sensors
• [ ] lvgl

[brghena]

I like this idea

[bradjc]

Note: I don't know where to put crc. In the kernel syscalls its under "Cryptography", but that seems not right.

[bradjc]

I've gone through and updated temperature and button to match the guide I wrote.

[tyler-potyondy]

@bradjc this looks great and does an excellent job formalizing the convention in guide.md. I'm in favor of these changes and think it would mark a good step forward towards having more accessible documentation.

[bradjc]

Couple things:

1. Should we namespace libtock functions with libtock_?
2. Should we put all header files in #include <libtock/sensors/temperature.h>? If so, how?
3. We probably need to specify the cpp wrapper for the header files.

[tyler-potyondy]

1. Should we namespace libtock functions with `libtock_`?

I am in support of this.

2. Should we put all header files in `#include <libtock/sensors/temperature.h>`? If so, how?

3. We probably need to specify the cpp wrapper for the header files.

Could you provide some more context with this. I'm not sure I follow with what this would be / how it would look.

[tyler-potyondy]

One other thought for the guide file:

There are some kernel assumptions regarding allow buffers. Although this is documented in the TRDs, I think we should provide a streamlined summary of how these buffers should and should not be used in this guide.

For instance (copied from the the syscall TRD):

The standard access model for allowed buffers is that userspace does not read or write a buffer that has been allowed: access to the memory is intended to be exclusive either to userspace or to the kernel. To regain access to a passed buffer B, the process calls the same Read-Write Allow system call again. If this call returns a success result, the result contains buffer B. The process can call with a zero-length buffer if it wishes to pass no memory to the kernel. Once a buffer has been returned to userspace as part of a Read-Write Allow system call, it is guaranteed for the kernel to no longer have access to the described memory region, unless it is currently shared with the kernel as part of the passed in buffer or another Allow mechanism.

Expecting users to call the allow_readwrite function again to regain control of the buffer seems somewhat clunky from the perspective of userspace. I think a wrapper within libtock-c may be worthwhile here (perhaps an unallow_readwrite or disallow_readwrite function).

Regardless, we should codify the expected convention and use in simple terms for this guide.

[bradjc]

What about putting all header files in libtock/include/libtock/[category]/? Then our apps would do #include <libtock/interface/led.h>.

This is what zephyr does: https://github.com/zephyrproject-rtos/zephyr/tree/main/include/zephyr

[alistair23]

The current method of including headers causes conflicts with other libraries. For example #include <alarm.h> is pretty common.

So putting it under libtock would be a great idea (the categories also help fix this issue)

[brghena]

While working on the redesign: is this the time to approach how to handle multiple instantiations of a single syscall driver?

Came up again in https://github.com/tock/tock/pull/3867

This could be handled with a struct for each driver that encapsulates its instantiated syscall number. Or it could be handled in the build system possibly, allowing us to include duplicate copies of a C file with a different #define value and name?

Or we could keep punting on it as an orthogonal problem that mostly occurs for downstream users (who could just make a duplicate C file with a different name if they wanted to).

[bradjc]

This could be handled with a struct for each driver that encapsulates its instantiated syscall number. Or it could be handled in the build system possibly, allowing us to include duplicate copies of a C file with a different #define value and name?

In these approaches, how would we default to what we have now? It seems like either we would have to add additional code in the app to choose the driver numbers, or apps could configure themselves in a makefile and completely change what they are doing with no changes to code. Neither seems desirable.

[bradjc]

We need to figure out an allow buffer management strategy. What layer is responsible for un-allowing the buffer? Are buffers passed back in callbacks? Or should users of the library hold their own copy of the pointer to the buffer?

[bradjc]

The unit test framework stores state. Where do we put it?

[bradjc]

Three more issues that have come up:

1. We need to use more of status_to_returncode
2. Should the upcall format be int? Why not just uint32_t?
3. Should we document that all tock.h functions start with tock_? What about subscribe(), etc.?

[bradjc]

I would also like to revisit what goes in examples/ and what goes in examples/tests.

[bradjc]

@tyler-potyondy @alevy Anything that would help with this?

[tyler-potyondy]

@tyler-potyondy @alevy Anything that would help with this?

Working on this now. Will push changes for UDP today and 15.4 tomorrow.

[bradjc]

Ping @alevy @hudson-ayers @tyler-potyondy

[hudson-ayers]

I am noticing that a lot of drivers have been ported but not the corresponding examples. what is the plan there?

[bradjc]

I added the list of apps that don't build (for me in my one test on 6b80b89c at least) to the main PR description.

The consensus on the call today is once this PR builds to go ahead and merge it.

[bradjc]

I edited the commits in this PR to remove duplicates and clean things up. Sorry if that messes with branches, but you should be able to just cherrypick any changes.

[bradjc]

I don't know what to do with UDP. The driver had some hidden state. I removed it so we can avoid warnings, but we need an actual fix.

[bradjc]

Ok we're down to just the networking and SPI test apps that do not compile.

[bradjc]

I added some documentation for libtock and libtock-sync/services.

I moved the printf/_write use of yield to libtock-sync. I moved read only state yield to libtock-sync. I moved unit test to libtock-sync services. The only remaining uses of yield in libtock are libraries which have not been updated.

[bradjc]

I cleaned up the commits.

I also fixed up the makefile -I flags so that apps have to use the libtock/ prefix in their includes.

I think the remaining todos are:

1. alarm
2. Run more tests on nrf52840dk. I have verified adc and console.

[bradjc]

I'm ready to merge this.

[ppannuto]

Pushed a tag for users who would like to defer updating to the new API here: https://github.com/tock/libtock-c/releases/tag/2.x-legacy-api — merging!