zephyr
zephyr copied to clipboard
POSIX Roadmap for LTSv3
Introduction
POSIX libraries and applications should Just Work™️ in 90% of the cases when being ported to Zephyr. Why? Because we want to enable our users to launch products and applications to the best of our ability, and we want to lower the barrier to entry by enabling users to leverage portable APIs.
This is an umbrella issue that will outlines work to be done in the POSIX API prior to LTSv3 (Ideally complete by v3.5).
Problem description
As discussed at EOSS 2023 and EOSS 2024, Zephyr's POSIX API layer needs some improvements. These can be broken down into the following:
- Maintainability
- Application / libc / toolchain Interface
- Application / library Portability
Proposed change
Implement all Option Requirements of POSIX Subprofiling Option Groups required for PSE51, PSE52, and most of PSE53 compatibility.
Update the POSIX areas at docs.zephyrproject.org. For reference, the top level page of Zephyr's POSIX documentation.
https://docs.zephyrproject.org/latest/services/portability/posix/index.html
Maintainability addresses the needs of the maintainer, contributors, and (to some extent) the users.
Interface addresses the needs of
- the user (users develop POSIX code to the interface, not the implementation) (i.e.
_POSIX_C_SOURCE,_XOPEN_SOURCE) - the C Library maintainers - via Option Requirements (e.g.
_POSIX_TIMERS) - users - we do not need to re-doc the standardized POSIX API (i.e. via doxygen). However, users should rightfully expect to be able to find links to the standards that Zephyr conforms to, as well as the conformance gaps.
Portability addresses the needs of the users who should (rightfully) expect that the POSIX API layer behaves according to the specification.
Detailed RFC
Proposed change (Detailed)
Maintainability
- Support toolchain / libc overrides for POSIX Option Groups
- [ ] https://github.com/zephyrproject-rtos/zephyr/pull/74184
- Abstract certain POSIX structures as integers
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/51781
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/52003
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/52087
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/52173
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/52313
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/52316
- Re-use Zephyr primitives inside of the POSIX API layer rather than dogfooding POSIX types
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/58924
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/56163
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/58856
- Remove layering violations / cyclic dependencies between
subsys/net,subsys/fs, andlib/posix- [x] https://github.com/zephyrproject-rtos/zephyr/issues/24966
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/73047
- [ ] https://github.com/zephyrproject-rtos/zephyr/pull/74096
- [ ] https://github.com/zephyrproject-rtos/zephyr/pull/73978
- Leverage a more standardized and compatible set of POSIX headers and Kconfig options
- PicoLibC (base on Newlib) headers already exist in Zephyr SDK
- Must copy-in for 3rd-party toolchains / libcs that do not support POSIX, so that Zephyr can still provide POSIX support
#include_nextis required for the above
- [ ] Insert link to PR for pulling in PicoLibc POSIX headers
- The POSIX Architecture and POSIX Subsystem are currently at odds
CONFIG_ARCH_POSIXandCONFIG_POSIX_APIare mutually exclusive- this is primarily due to conflicting symbols in the C global namespace
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/13787
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/58305
- [x] #45100
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/67181
- Remove
CONFIG_NET_SOCKETS_POSIX_NAMES- introduced prior to
CONFIG_POSIX_APIin spite of the fact that the BSD Sockets API is part of POSIX
- [ ] TODO insert PR for removing net sockets posix names
- introduced prior to
The layering issues that exist between subsys/net and lib/posix can be described as a cyclic dependency. The issue can be resolved via mutual dependency.
I.e. Currently, we have:
┌───────────┐ ┌───────────┐
│ ├──────►│ │
│ POSIX │ │ NET │
│ │◄──────┤ │
└───────────┘ └───────────┘
But we need
┌───────────┐
│ │
│ ZVFS │
│ │
└▲────────▲─┘
│ │
│ │
┌──────────┴┐ ├───────────┐
│ │ │ │
│ POSIX ├───────► NET │
│ │ │ │
└───────────┘ └───────────┘
Diagrams made with https://asciiflow.com/
I think we may have settled on the name "ZVFS" in the above diagram.
Interface
[!TIP] Check the Current status of Suprofiling Option Group support from docs to verify that the items below are up to date
- [x] Include a clear reference to the POSIX standard in Zephyr's documentation
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/64540
- Plan for formally supporting Real-Time and Embedded Profiles of IEEE Std 1003.13-2003 (a.k.a. PSE52)
- https://github.com/zephyrproject-rtos/zephyr/issues/17706
- Conformance Test Suites:
- VSX4 POSIX IEEE 1003.1 (Note: test is designed for IEEE 1003.1-2001)
- VSTH Threads
- VSRT Real-Time
- With
CONFIG_POSIX_API, Ensure POSIX headers can be included as#include <poll.h>rather than#include <zephyr/posix/poll.h>- [x] https://github.com/zephyrproject-rtos/zephyr/pull/51771
POSIX_SINGLE_PROCESS- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66866
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66861
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66862
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66863
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/56670
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59924
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66864
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/66762
POSIX_SIGNALS- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59925
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59927
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59928
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59929
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59931
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59932
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59933
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66923
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66924
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66925
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66928
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66929
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66930
POSIX_DEVICE_IO- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66932
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66938
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66946
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66947
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66948
POSIX_FD_MGMT- [x] https://github.com/zephyrproject-rtos/zephyr/issues/74097
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/74098
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/74099
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/74100
POSIX_SEMAPHORES- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66955
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66956
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66957
_POSIX_MESSAGE_PASSING- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66958
_POSIX_PRIORITY_SCHEDULING- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66959
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66960
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66961
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66962
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66963
_POSIX_READER_WRITER_LOCKS- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66964
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66965
_POSIX_THREAD_PRIORITY_SCHEDULING- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66966
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66967
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66968
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66969
_POSIX_TIMEOUTS- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66970
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66971
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/66972
_XOPEN_STREAMS- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66973
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66974
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66975
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66976
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66977
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66978
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/66979
_POSIX_SPIN_LOCKS(mandatory for the base specification)- [x] https://github.com/zephyrproject-rtos/zephyr/pull/59911
- [x] posix: implement
pthread_spin_destroy() - [x] posix: implement
pthread_spin_init() - [x] posix: implement
pthread_spin_lock() - [x] posix: implement
pthread_spin_trylock() - [x] posix: implement
pthread_spin_unlock() - [x] https://github.com/zephyrproject-rtos/zephyr/issues/59910
_POSIX_THREADS- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59934
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59935
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59936
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59937
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59939
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59940
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59941
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59942
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59943
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59944
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59945
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59946
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/44379
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/59773
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/44727
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/25973
_POSIX_CLOCK_SELECTION- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59947
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59948
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59949
_POSIX_SHARED_MEMORY_OBJECTS- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59950
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59951
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59952
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59953
_POSIX_CPUTIME- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59954
_POSIX_TIMERS- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59955
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/59956
- https://github.com/zephyrproject-rtos/zephyr/issues/40099
- https://github.com/zephyrproject-rtos/zephyr/pull/17155
_POSIX_REALTIME_SIGNALS- [x] https://github.com/zephyrproject-rtos/zephyr/issues/74993
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/74994
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/74995
_POSIX_SYNCHRONIZED_IO- [x] implement
fsync() - [x] implement
msync() - [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74455
- [x] implement
_POSIX_THREAD_PRIO_PROTECT- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74456
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74457
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74458
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74459
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74460
_POSIX_THREAD_SAFE_FUNCTIONS- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74461
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74462
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74463
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74464
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74465
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74466
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74467
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74468
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74469
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74470
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74471
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74472
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74473
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74474
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/74475
- Address regular bugs and enhancement requests
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/25407
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/51776
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/52360
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/52517
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/52518
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/52749
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/55186
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/54996
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/58911
- [x] https://github.com/zephyrproject-rtos/zephyr/issues/54996
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/46101
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/60081
Dependencies
Concerns and Unresolved Questions
Alternatives
There was a previous massive PR that attempted to address a number of the goals at once (#43987), but it is impossible to review in the current state and needs to be broken down and simplified in separate individual PRs.
Architecture WG:
- @cfriedt shows a presentation with a roadmap update for POSIX support
- @nashif asks about the scope of POSIX support, which profiles will be supported? @cfriedt answers that so far he's focused on stabilizing and fixing features instead of adding new ones. Zephyr would benefit from having a documentation page similar to: https://docs.rtems.org/branches/master/posix-compliance/standards.html
- @cfriedt: when ever implementing dynamic features in the scheduler / core code (not yet defined safety scope) please consider safety where dynamic memory management and using heaps is problematic to get TÜV certification. But I remember that we had that discussion earlier already and solved that by layering the dynamic part on top of the core code which can be passed statically OR dynamically allocated objects such as stack buffers, thread contexts etc. what so ever. I feel you are aware of that.
- @cfriedt: when ever implementing dynamic features in the scheduler / core code (not yet defined safety scope) please consider safety where dynamic memory management and using heaps is problematic to get TÜV certification.
Absolutely
But I remember that we had that discussion earlier already and solved that by layering the dynamic part on top of the core code which can be passed statically OR dynamically allocated objects such as stack buffers, thread contexts etc. what so ever. I feel you are aware of that.
Yeah, so the suggested approach from an earlier PR was to keep thread stack allocation separate and selectable, so that will still be the case. However, the proposed kernel-level API will also allow
- a static, pre-allocated pool of thread stacks to be used
- dynamic, heap-allocated thread stacks to be used
- both of the above (with a configurable preference)
- and of course neither
Yeah, so the suggested approach from an earlier PR was to keep thread stack allocation separate and selectable, so that will still be the case. However, the proposed kernel-level API will also allow - a static, pre-allocated pool of thread stacks to be used - dynamic, heap-allocated thread stacks to be used - both of the above (with a configurable preference) - and of course neither
Like it!
Would be kind of nice to be able to use socketpair() without networking. I wonder if something can be borrowed from eventfd.. 🤔
@cfriedt speaking of POSIX PSE profile test-suites (in the original issue description above), it might be a bit more correct to refer to "embedded versions" of those, such as:
@cfriedt you updated names. but references are still to non-PSE pages ;)
@cfriedt you updated names. but references are still to non-PSE pages ;)
Actually, all three are referenced from "Conformance Test Suites" already. I was just trying to provide a reference to the higher scope specifications as well.
In any case, the information is available.
TSC Attention Required
Please refer to
POSIX System Interfaces POSIX Application Environment Profiles
Why should the PRs below be considered for approval after Feature Freeze?
- A number of members of the Zephyr project (especially those in automotive, other safety-centric areas), depend on POSIX for product development for the next several years.
- Many features were part of the release plan.
- Not all features were finished on time, but a huge amount were, and some had approvals but did not get merged in time
- Better to not leave architecturally important PRs on the table
- Some instability without
POSIX_DEVICE_IO,POSIX_FD_MGMT POSIX_FD_MGMT,POSIX_DEVICE_IOwould likely be required for libc bugfixes (properly utilizing zvfs layer).
What are the PR's we are asking to merge?
Allow toolchains / platforms to override POSIX Option Groups via Kconfig (mainly for maintainers, libc compatibility)
- [x] https://github.com/zephyrproject-rtos/zephyr/pull/74184
Add stubs for the POSIX_SIGNALS Option Group (i.e. signalling processes. We do not yet have processes in Zephyr, so stubs for functions that are simply not supportable)
- [ ] https://github.com/zephyrproject-rtos/zephyr/pull/74436
Implement POSIX_DEVICE_IO Option Group (depended on mmap, so a bit delayed. lays groundwork for zvfs, extremely useful in terms of reducing api-layer dependency cycles)
- [ ] https://github.com/zephyrproject-rtos/zephyr/pull/73978
Implement POSIX_FD_MGMT Option Group (zvfs_poll(), zvfs_select(), extremely useful in terms of reducing api-layer dependency cycles)
- [ ] https://github.com/zephyrproject-rtos/zephyr/pull/74096
What are we leaving behind?
System Inferfaces (Base Definitions)
For most practical purposes, the System interfaces are feature complete 🎉(with caveats).
-
[ ] https://github.com/zephyrproject-rtos/zephyr/pull/74918
_POSIX_REALTIME_SIGNALSalthough tests are passing, this was rejected by the TSC since it did not make the feature freeze cutoff, because it had not been fully reviewed, and because more documentation was requested- LTSv3 users should still be able to add this support themselves via the patch
- includes the one missing piece of the
_POSIX_THREADSpuzzle (pthread_kill())
-
[ ] https://github.com/zephyrproject-rtos/zephyr/pull/74350
-
[ ] https://github.com/zephyrproject-rtos/zephyr/pull/74180
-
[ ] https://github.com/zephyrproject-rtos/zephyr/pull/74394
_POSIX_THREAD_SAFE_FUNCTIONSvery close to being done, but still needs review time, has dependencies on posix device io, etc- revealed a number of issues in libc integration
- several of these functions have to do with unsupported features (user & group id's, etc)
- i.e. many functions would just be stubs anyway
PSE51 - Minimal Realtime System Profile
For most practical purposes, PSE51 is feature complete 🎉(with caveats).
- Most functionality is there as far as can be supported today in Zephyr
- e.g. Zephyr does not support metadata synchronization, mutex priorities, or a sporadic thread scheduler
- [ ] https://github.com/zephyrproject-rtos/zephyr/pull/74383 (adds stub for
fdatasync(), so a conformant app can still link) - [ ] https://github.com/zephyrproject-rtos/zephyr/pull/74386 (adds stubs, so a conformant apps can still link)
- [ ]
_POSIX_THREAD_SPORADIC_SERVER(missingSCHED_SPORADICconstant)
PSE52 - Realtime Controller System Profile
POSIX_FILE_SYSTEMsubsys/fsrequires betterfdtableintegration (i.e. using per-file-descriptor offset, rather than per-file offset)- depends on device io, fd mgmt, etc
- some missing calls (mostly wrappers around existing native zephyr FS API calls)
_POSIX_TRACE*- not a lot of demand, as Zephyr already has a pretty good tracing subsystem
- had considered implementing as stubs, but not enough cycles
PSE53 - Dedicated Realtime System Profile
- quite a few things relating to multi-process support (pipe, signal jump, spawn, sporadic server)
- would rather just do this properly after v3.7.0 instead of just introducing stubs.
Why did some features not make it in for Feature Freeze?
- Several features depend on process support, which does not (yet) exist in Zephyr.
- Dependencies spread out over multiple PRs, libc's, kernel, net, fs, arch.
- Difficulty making parallel PRs (PR dependencies = serial PRs, which takes longer)
- Limited developer and reviewer bandwidth. Big thank you to project members who were kind enough to review POSIX PRs!🙏
- Some Options in particular are beyond the scope of what is supported in Zephyr, so would only be stubs (e.g. mutex priorities)
Post-freeze Bug-fixes and Stability-fixes (possible backports)
- [ ] DOCUMENTATION We definitely need better API-level documentation, Doxygen, etc
- [ ] Optional POSIX features should be guarded by feature test macros (e.g.
#if defined(_POSIX_PRIORITY_SCHEDULING)) - [ ] Use correct header names (e.g.
sys/features.hinstead ofposix_features.h - [ ] Mitigate type definition conflicts between Newlib / Picolibc / Minimal libc
- [ ] Merge
signal.handtime.hinto minimal libc to reduce conflicts between Newlib / Picolibc / Minimal libc - [ ] Do NOT blanket-define
_POSIX_THREADS(lib/posix/options/CMakeLists.txt) - [ ] https://github.com/zephyrproject-rtos/zephyr/pull/71832
Updated comment to reflect that the the posix device_io, signals, and fd_mgmt features were reverted.
This puts LTSv3 in a kind of dilapidated state from the POSIX & libc perspective.