bolt Stream only some things

Stream only some things

Open rlpowell opened this issue 2 years ago • 7 comments

Use Case

--stream is generally great and fantastic, but it becomes a lot less fantastic when dealing with tasks that return structured data. In my project I have a mix of tasks that do stuff whose output I want to see, and tasks that return structured data for processing by the plan, where I very much don't. Especially when there's pages and pages of said structured data. See a minimal example at the end of this issue

Describe the Solution You Would Like

Honestly I'm not sure what a great solution looks like. I think run_task should have a flag that says "this task definitely returns structured data", both so it could be ignored by stream and so that the plan would fail if no valid json was returned. Having a flag for run_task, run_plan, and run_command that allowed the explicit disabling (or enabling!) of stream would also be great.

In fact, while I think the first thing is a good idea regardless, I prefer the second solution so that the plan writer can stream things or not as they choose for best readability for other users of the plan in the future.

Describe Alternatives You've Considered

I can't think of any other ways to handle this; the current behaviour is definitely pretty unpleasant.

Additional Context

Here's an example to showcase the problem; in my real project there are tasks like this one but also tasks I very much want to see the in-progress output of.

plan myproject::test_stream(
) {
  $json_results = run_task('myproject::test_stream_2', 'localhost')
  unless $json_results.ok {
      out::message($json_results)
      fail_plan('Failed to generate some json')
  }

  $result=$json_results.first().value()['a'][17]
  out::message("Result: $result")
}

test_stream_2.py:

#!/usr/bin/env python3

import json
import sys

nums=list(range(12,500))
json.dump({ 'a': nums, 'b': nums}, sys.stdout)

Normal output:

# bolt plan run myproject::test_stream
Starting: plan myproject::test_stream
Starting: task myproject::test_stream_2 on localhost
Finished: task myproject::test_stream_2 with 0 failures in 0.07 sec
Result: 29
Finished: plan myproject::test_stream in 0.08 sec
Plan completed successfully with no result

Stream output, which you'll notice is both long but also formatted really weirdly; the actual program output is all on one line:

# bolt plan --stream run myproject::test_stream
Starting: plan myproject::test_stream
Starting: task myproject::test_stream_2 on localhost
The 'stream' option is experimental, and might include breaking changes between minor versions. [ID: stream_experimental]
[localhost] out: {"a": [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25
[localhost] out: , 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81
[localhost] out: , 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101
[localhost] out: , 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114
[localhost] out: , 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131
[localhost] out: , 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143
[localhost] out: , 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157
[localhost] out: , 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180
[localhost] out: , 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196
[localhost] out: , 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211
[localhost] out: , 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224
[localhost] out: , 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236
[localhost] out: , 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250
[localhost] out: , 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262
[localhost] out: , 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284
[localhost] out: , 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298
[localhost] out: , 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311
[localhost] out: , 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324
[localhost] out: , 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336
[localhost] out: , 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352
[localhost] out: , 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372
[localhost] out: , 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385
[localhost] out: , 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398
[localhost] out: , 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424
[localhost] out: , 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437
[localhost] out: , 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450
[localhost] out: , 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463
[localhost] out: , 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476
[localhost] out: , 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488
[localhost] out: , 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499]
[localhost] out: , "b": [12, 13, 14, 15, 16, 17, 18, 19, 20
[localhost] out: , 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33
[localhost] out: , 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45
[localhost] out: , 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58
[localhost] out: , 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72
[localhost] out: , 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100
[localhost] out: , 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115
[localhost] out: , 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129
[localhost] out: , 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142
[localhost] out: , 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155
[localhost] out: , 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168
[localhost] out: , 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180
[localhost] out: , 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195
[localhost] out: , 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208
[localhost] out: , 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220
[localhost] out: , 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233
[localhost] out: , 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246
[localhost] out: , 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259
[localhost] out: , 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282
[localhost] out: , 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297
[localhost] out: , 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309
[localhost] out: , 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322
[localhost] out: , 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335
[localhost] out: , 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348
[localhost] out: , 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361
[localhost] out: , 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374
[localhost] out: , 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400
[localhost] out: , 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414
[localhost] out: , 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426
[localhost] out: , 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438
[localhost] out: , 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451
[localhost] out: , 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463
[localhost] out: , 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475
[localhost] out: , 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488
[localhost] out: , 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499]
[localhost] out: }
Finished: task myproject::test_stream_2 with 0 failures in 0.07 sec
Result: 29
Finished: plan myproject::test_stream in 0.09 sec
Plan completed successfully with no result
#

Mar 16 '22 23:03 rlpowell

Why does it wrap lines, but wrap different lines at different places? So bizarre. :D

Mar 16 '22 23:03 rlpowell

@rlpowell Bolt receives output in chunks, and immediately prints those chunks to the console as it receives them. That's why the lines wrap and wrap in different places -- Bolt is just receiving different size chunks of data.

As for being able to toggle streaming for only specific tasks, we can discuss as a team if that is something we want to explore. However, considering the feature is experimental, I'm not sure that it will be prioritized at this time.

Jun 01 '22 23:06 beechtom

I'd like to have _stream parameter of run_* functions which allows or disallows the streaming from the function.. or maybe extend without_default_logging() to streaming as well (if it's not done yet).

This way one can exclude some parts from streaming.