componentize-py
componentize-py copied to clipboard
bytecodealliance
pydantic support
did anyone try pydantic with componentize-py?
I ran into this issue and was told to raise an issue here.
I am not able to compile the code to a wasm binary error:
(.venv) (base) wasm/component-reconciler - (result) > just build-guest-python
/opt/homebrew/bin/uv
==> building python guest component...
Traceback (most recent call last):
File "/Users/henderiw/.cache/uv/archive-v0/_oAvmxPnx7MxwA070F1Ji/bin/componentize-py", line 10, in <module>
sys.exit(script())
~~~~~~^^
AssertionError: Traceback (most recent call last):
File "/0/rec.py", line 3, in <module>
from apis.topo.v1alpha1.topology_types import Topology
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/0/apis/topo/v1alpha1/topology_types.py", line 1, in <module>
from pydantic import BaseModel, Field, ValidationError, model_validator
File "/1/pydantic/__init__.py", line 421, in __getattr__
module = import_module(module_name, package=package)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/python/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/1/pydantic/main.py", line 29, in <module>
import pydantic_core
File "/1/pydantic_core/__init__.py", line 6, in <module>
from ._pydantic_core import (
ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'
Caused by:
ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'
I installed the modules in my venv (.venv) (base) wasm/component-reconciler - (result) > uv pip show typing-extensions Name: typing-extensions Version: 4.12.2 Location: /Users/henderiw/code/wasm/component-reconciler/.venv/lib/python3.11/site-packages Requires: Required-by: pydantic, pydantic-core (.venv) (base) wasm/component-reconciler - (result) > (.venv) (base) wasm/component-reconciler - (result) > uv pip show pydantic_core Name: pydantic-core Version: 2.27.2 Location: /Users/henderiw/code/wasm/component-reconciler/.venv/lib/python3.11/site-packages Requires: typing-extensions Required-by: pydantic Not sure what is going on? (edited)
the code with the error is in this PR
https://github.com/henderiw/component-reconciler/pull/3
Hi @henderiw. Thanks for reporting this.
I just tried making a simple app that uses Pydantic, and it seems to work. Here's what I did:
mkdir foo
cd foo
python3 -m venv .venv
source .venv/bin/activate
cargo install wasmtime-cli
pip install pydantic==2.5.3 componentize-py
curl -OL https://github.com/WebAssembly/wasi-cli/archive/refs/tags/v0.2.2.tar.gz
tar xf v0.2.2.tar.gz
curl -OL https://github.com/dicej/wasi-wheels/releases/download/latest/pydantic_core-wasi.tar.gz
tar xf pydantic_core-wasi.tar.gz
cat >app.py <<EOF
from command import exports
from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel
class User(BaseModel):
id: int
name: str = 'John Doe'
signup_ts: Optional[datetime] = None
friends: List[int] = []
class Run(exports.Run):
def run(self) -> None:
external_data = {'id': '123', 'signup_ts': '2017-06-01 12:22', 'friends': [1, '2', b'3']}
user = User(**external_data)
print(user)
EOF
componentize-py -d wasi-cli-0.2.2/wit -w command componentize app -o cli.wasm
wasmtime run cli.wasm
Note that pydantic depends on pydantic-core, which is written in Rust and compiled as a native extension. When you run pip install pydantic, pip will download and install a copy of pydantic-core for your native platform (e.g. MacOS/ARM64 or whatever) -- not wasm32-wasi, which is what we actually want. And since there's not yet an official way to publish WASI wheels for Python, we must use a build from the wasi-wheels repo instead. And since that build is about a year old now, we need to also use an old pydantic package which matches (v2.5.3). If you want to use the latest version, you'll need to build the corresponding version of pydantic-core for wasm32-wasi yourself (e.g. using this build script).
BTW, @benbrandt is working on improving the WASI wheel situation. The goal is to create an "official" package index to which you'll be able to point pip so it can download and install WASI wheels itself. That will remove the need to download and unpack a .tar.gz file manually.
Hope that helps!
I can now build indeed the component. but when I run it I get this error
(.venv) (base) wasm/component-reconciler - (result) > just run-guest-python
==> running python guest component...
Finished dev profile [unoptimized + debuginfo] target(s) in 0.10s
Running target/debug/reconciler
Running iteration: 1
Traceback (most recent call last):
File "/0/rec.py", line 15, in reconcile
File "/1/pydantic/_internal/_model_construction.py", line 92, in new
File "/1/pydantic/_internal/model_construction.py", line 308, in inspect_namespace
ModuleNotFoundError: No module named 'pydantic.fields'
thread 'RUST_BACKTRACE=1 environment variable to display a backtrace
Reconcile Iteration 0 failed: ReconcileError { code: 500, message: "Failed to call reconcile: error while executing at wasm backtrace:\n 0: 0xc1a397 - libcomponentize_py_runtime.so!__rust_start_panic\n 1: 0xc1a208 - libcomponentize_py_runtime.so!rust_panic\n 2: 0xc1a1db - libcomponentize_py_runtime.so!std::panicking::rust_panic_with_hook::h683e04a3e610de41\n 3: 0xc1263d - libcomponentize_py_runtime.so!std::panicking::begin_panic_handler::
this is the code I used
import reconciler
from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel
class Reconciler(reconciler.Reconciler):
def reconcile(self, object: str) -> reconciler.ReconcileResult:
# Example values for the result
requeue = False # Whether to requeue
requeue_after = 30 # Requeue after 30 seconds
#response_object = f"Processed: {object}" # Return a string with processed information
class User(BaseModel):
id: int
name: str = 'John Doe'
signup_ts: Optional[datetime] = None
friends: List[int] = []
external_data = {'id': '123', 'signup_ts': '2017-06-01 12:22', 'friends': [1, '2', b'3']}
user = User(**external_data)
print(user)
# Return the result with appropriate values
return reconciler.ReconcileResult(
requeue=requeue,
requeue_after=requeue_after,
object=f"Processed: {object}"
)
without the pydantic code it runs fine
import reconciler
class Reconciler(reconciler.Reconciler):
def reconcile(self, object: str) -> reconciler.ReconcileResult:
# Example values for the result
requeue = False # Whether to requeue
requeue_after = 30 # Requeue after 30 seconds
# Return the result with appropriate values
return reconciler.ReconcileResult(
requeue=requeue,
requeue_after=requeue_after,
object=f"Processed: {object}"
)
(.venv) (base) wasm/component-reconciler - (result) > just run-guest-python
==> running python guest component...
Finished dev profile [unoptimized + debuginfo] target(s) in 0.09s
Running target/debug/reconciler
Running iteration: 1
Reconcile Iteration 0 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 0 elaspetime 1.548375ms
Running iteration: 2
Reconcile Iteration 1 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 1 elaspetime 24.042µs
Running iteration: 3
Reconcile Iteration 2 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 2 elaspetime 15.917µs
Running iteration: 4
Reconcile Iteration 3 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 3 elaspetime 19.875µs
Running iteration: 5
Reconcile Iteration 4 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 4 elaspetime 14.584µs
Running iteration: 6
Reconcile Iteration 5 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 5 elaspetime 11.875µs
Running iteration: 7
Reconcile Iteration 6 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 6 elaspetime 12.875µs
Running iteration: 8
Reconcile Iteration 7 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 7 elaspetime 18.625µs
Running iteration: 9
Reconcile Iteration 8 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 8 elaspetime 11.667µs
Running iteration: 10
Reconcile Iteration 9 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
I can also confirm your example works for me. but in my runtime it doesn't.
this is my current runtime for reference
use std::path::PathBuf;
use anyhow::{Context, Result};
use wasmtime::component::{bindgen, Component, Linker};
use wasmtime::{Engine, Store};
use wasmtime_wasi::{ResourceTable, WasiCtx, WasiView};
use std::time::Instant;
bindgen!({path: "../../../wit", world: "reconciler", async: false});
/// This state is used by the Runtime host,
/// we use it to store the WASI context (implementations of WASI)
/// and resource tables that components will use when executing
///
/// see:
/// - https://docs.rs/wasmtime-wasi/latest/wasmtime_wasi/trait.WasiView.html
/// - https://docs.rs/wasmtime-wasi/latest/wasmtime_wasi/fn.add_to_linker_sync.html
struct HostState {
ctx: WasiCtx,
table: ResourceTable,
}
impl HostState {
pub fn new() -> Self {
let ctx = WasiCtx::builder().inherit_stdio().build();
Self {
ctx,
table: ResourceTable::default(),
}
}
}
impl ReconcilerImports for HostState {
fn get(&mut self, name: String) -> String {
println!("Host received name: {}", name);
format!("Hello, {}!", name)
}
}
impl WasiView for HostState {
fn ctx(&mut self) -> &mut WasiCtx {
&mut self.ctx
}
fn table(&mut self) -> &mut ResourceTable {
&mut self.table
}
}
/// load the WASM component and return the instance
fn load_reconciler_instance(
path: PathBuf,
) -> Result<(Store<HostState>, Reconciler)> {
// Initialize the Wasmtime engine
let engine = Engine::default();
// Load the WASM component
let component = Component::from_file(&engine, &path).context("Component file not found")?;
// Create the store to manage the state of the component
let states: HostState = HostState::new();
let mut store = Store::<HostState>::new(&engine, states);
// Set up the linker for linking interfaces
let mut linker = Linker::new(&engine);
// Add WASI implementations to the linker for components to use
wasmtime_wasi::add_to_linker_sync(&mut linker)?;
// Add the `Reconciler` interface to the linker
Reconciler::add_to_linker(&mut linker, |state| state)?;
// Instantiate the component
let instance = Reconciler::instantiate(&mut store, &component, &linker)
.context("Failed to instantiate the reconciler world")?;
Ok((store, instance))
}
// call the reconcile function
fn call_reconcile(
store: &mut Store<HostState>,
instance: &Reconciler,
input_json: String,
) -> std::result::Result<ReconcileResult, ReconcileError> {
// Call the reconcile function
let result = instance
.call_reconcile(store, &input_json)
.map_err(|e| ReconcileError {
code: 500,
message: format!("Failed to call reconcile: {}", e),
})??;
Ok(result)
}
fn main() -> Result<()> {
let wasm_path = PathBuf::from(
std::env::var_os("GUEST_WASM_PATH")
.context("missing/invalid path to WebAssembly module (env: GUEST_WASM_PATH)")?,
);
// Input JSON
let input_json = r#"{"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}"#.to_string();
//load the instance
let (mut store, instance) = load_reconciler_instance(wasm_path)
.map_err(|e| anyhow::anyhow!("Error loading reconciler instance: {}", e))?;
// Measure time taken to run the instance 10 times
let start = Instant::now();
for i in 0..10 {
println!("Running iteration: {}", i + 1);
// Measure iteration time
let iteration_start = Instant::now();
match call_reconcile(&mut store, &instance, input_json.clone()) {
Ok(result) => {
let iteration_duration = iteration_start.elapsed();
println!("Reconcile Iteration {} succeeded with output: {:#?}", i, result);
println!("Reconcile Iteration {} elaspetime {:?}", i, iteration_duration);
}
Err(e) => {
let iteration_duration = iteration_start.elapsed();
eprintln!("Reconcile Iteration {} failed: {}", i, e);
println!("Reconcile Iteration {} elaspetime {:?}", i, iteration_duration);
}
}
}
let duration = start.elapsed();
println!("Time taken for 10 iterations: {:?}", duration);
Ok(())
}
ModuleNotFoundError: No module named 'pydantic.fields'
I suspect this is related to the issue described here and here. I would recommend moving the class User(BaseModel): declaration to the top level of the file. Another option might be to add a from pydantic import fields to the top level of the file (and iterate if there are other modules which need to be explicitly pulled in).
confirmed.
this works
import reconciler
from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel
class User(BaseModel):
id: int
name: str = 'John Doe'
signup_ts: Optional[datetime] = None
friends: List[int] = []
class Reconciler(reconciler.Reconciler):
def reconcile(self, object: str) -> reconciler.ReconcileResult:
# Example values for the result
requeue = False # Whether to requeue
requeue_after = 30 # Requeue after 30 seconds
external_data = {'id': '123', 'signup_ts': '2017-06-01 12:22', 'friends': [1, '2', b'3']}
user = User(**external_data)
print(user)
# Return the result with appropriate values
return reconciler.ReconcileResult(
requeue=requeue,
requeue_after=requeue_after,
object=f"Processed: {object}"
)
Running iteration: 1 id=123 name='John Doe' signup_ts=datetime.datetime(2017, 6, 1, 12, 22) friends=[1, 2, 3] Reconcile Iteration 0 succeeded with output: ReconcileResult { requeue: false, requeue-after: 30, object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}", } Reconcile Iteration 0 elaspetime 3.163375ms