Async documentation

[bruno-]

Hi,

@ioquatix as we talked on the call the other day, here's my list of questions about async that I wish I could have read somewhere so I can get up to speed faster. I'll add question for any async-* gem, not just core async.

Anyone can add their own questions. These will be used when building the docs for async gems.

Note: I don't expect answers to these questions. I just think it would be useful to document this stuff.

Related: here's another ruby async project that has decent webpage and docs https://digital-fabric.github.io/polyphony/

async gem

• What is a "task"? How is it different from fiber? Why does async manipulate tasks and not raw fibers?

• What are transient tasks?

• What really are fibers?

  • The answer "a cooperative scheduling concurrency mechanism" is really not helpful.
  • This talk from C++ community really helped me better understand this concept https://www.youtube.com/watch?v=_fu0gx-xseY. Maybe we can draw notes from that talk when explaining fibers?

• What is a selector?

• What is a reactor? How is it different from selector?

• When using async should I make any method be asynchronous? Example, should I wrap any method in async block like thid: Async::Task.current.async { print_to_logger }. Also related to async-await gem: should I define all the methods with async def my_method.

• The readme shows using barrier and semaphore together. Do I really always need to use these two together?

• Question related to semaphore and barrier use:

barrier = Async::Barrier.new
semaphore = Async::Semaphore.new(2)

# Example 1
semaphore.async(parent: barrier) do
  # ...
end

# Example 2
barrier.async(parent: semaphore) do
  # ...
end

Which of the two examples above is right? If both are right, when do I use one and when do I use the other?

• (maybe bug?) I would like more info on how #every, #after and #sleep methods work. Here are a couple examples I found to be contradictory:

require 'async'

Async do |task| # reactor exits immediately, not what I expected
  task.reactor.after(1) do
    puts "Example 1" # never runs, not sure why
  end
end

Async do |task|
  task.reactor.sleep(1)
  puts "Example 2" # this one runs, all good
end

Async do |task|
  task.async do |subtask|
    subtask.sleep 1
    puts "Example 3" # this one runs, all good
  end
end

Async do |task| # reactor exits immediately, not what I expected
  task.reactor.every(1) do
    puts "Example 4" # never runs, I expect this to run forever
  end
end

• (maybe an enhancement) Can we have Async::Task#every and Async::Task#after method?

async-io

• What's the difference between a io, socket, endpoint and a stream? They're all similar concepts, but they're used differently.

• Why don't you just monkey patch core ruby classes? Why did you go with wrappers? What are the monkey patching pros and cons?

async-http

• Why is Async::HTTP::Internet the main class for making requests? How does it differ from Async::HTTP::Client? Other http libraries use "client" for making requests, why doesn't async do that?

Questions related to all async gems

• Can I use async with rails?

• I'm at the point where I know how to use async suite of gems productively. But I'm curious about the magic behind making async http requests. How can you trigger a http request and continue running other ruby code? Isn't ruby supposed to block after a http request is made? Answer suggestion: it took me a lot of time to figure this one out. Reading these methods was helpful in understanding how the magic works:

  1. https://github.com/socketry/async-io/blob/6835c9853f370c7e8a190a1baea4990c3aef7043/lib/async/io/generic.rb#L57-L72 (also see how and where wrap_blocking_method is used)
  2. https://github.com/socketry/async-io/blob/cfbf3891379240871cd91db2be442948b5a05c6a/lib/async/io/generic.rb#L214-L227
  3. https://github.com/socketry/async/blob/17ef44fb0e0bedadcdc96dce60fc4f35d74ac6cc/lib/async/wrapper.rb#L117-L129
  4. https://github.com/socketry/async/blob/17ef44fb0e0bedadcdc96dce60fc4f35d74ac6cc/lib/async/wrapper.rb#L132-L144

[jsaak]

I want to integrate a non-blocking C API with the Reactor. I found Reactor#register, i think it may be able to do it, but i have no idea how. I working example would be delightful. WITH error handling. Do not just ignore error handling, since that is the most difficult part with these tasks usually.

[trevorturk]

Just a note to say that https://github.com/socketry/async-examples has a bunch of example code which may be useful to people just getting started with Async, especially when used in conjunction with GraphQL.

[jsaak]

How do you implement a function like Open3.capture3() ? Using open3 does not seem to work, however it is a part of the standard library!

This does not work either:

  pipe_error_r, pipe_error_w = IO.pipe
  pipe_error_w.close
  pipe_stdout_r, pipe_stdout_w = IO.pipe
  pipe_stdout_w.close
  
  pid = spawn(commandline, :err => pipe_error_r, :out => pipe_stdout_r)

  err = ''
  Fiber.schedule do
    while line = pipe_error_r.gets do
      err += line
    end
  end

  stdout = ''
  Fiber.schedule do
    while line = pipe_stdout_r.gets do
      stdout += line
    end
  end

  pid, status = Process.wait2 pid
  
  if status.exitstatus != 0
      puts "exitcode: #{status.exitstatus}, stderr: #{err}"
  end

[ioquatix]

You will need to put the entire thing into a Fiber if you want Process.wait2 to be non-blocking and allow other fibers to execute.

[daipom]

Thanks for the documentation!

I saw Getting Started, and I felt the examples of Creating an Asynchronous Tasks and Waiting for Results are misleading.

https://github.com/socketry/async/blob/c5911ec9b0e4dca9896cf0b4063d71c77e5a614e/guides/getting-started/readme.md?plain=1#L21-L47

When I saw these examples, I understood that the blocks being passed to the Async run asynchronously. However, it was wrong and the blocks run synchronously, right? (The top-level Async is executed synchronously, not Asynchronously)

I'm not sure what would be a better example of asynchronous execution in Getting Started, but I'm just sharing what I felt after reading Getting Started for now.

[ioquatix]

I believe most of these questions (at least, the async specific ones) are covered by the updated documentation.

However, there are some specific questions I'll answer here.

Question related to semaphore and barrier use (the order)

I believe it's better to put the barrier at the end of the chain. However, I don't think it matters very much in practice.

I would like more info on how #every, #after and #sleep methods work.

Those methods are now removed and shouldn't be used, they were dangerous because they ran as bare timers. We will probably eventually remove the dependency on the timers gem too.

Why don't you just monkey patch core ruby classes? Why did you go with wrappers? What are the monkey patching pros and cons?

Async 2+ uses hooks implemented in the Ruby language and the wrappers are no longer required.

How do you implement a function like Open3.capture3()?

In Async 2+, you can use Open3 without any special consideration and it is internally non-blocking.

[ioquatix]

Also, sorry for replying 3 years later :)

[ioquatix]

A few more points about async-io:

• It's almost done it's dash. Once Async 1.x is EOL, async-io will almost certainly be EOL too. We will instead use Ruby's IO or IO::Buffer for the same purpose.
• Async::IO::Endpoint is going to be replaced with IO::Endpoint which won't depend on Async hopefully and will be compatible with other fiber schedulers. It's taken a bit of time to get all the interfaces in core Ruby, e.g. IO#timeout were required.