string output stream?

[Zulu-Inuoe]

Hey all,

One feature I tend to miss often is the ability to have a stream output to a string with a fill-pointer ala with-output-to-string, but with indefinite extent so I can keep it around.

I see this is missing even from flexi-streams, with the same issue where you have flexi-streams:with-output-to-sequence, but no way to make-output-to-sequence.

Any thoughts on an addition like that? I was going to write my own function making use of flexi-streams::vector-output-stream, as that's what make-in-memory-output-stream uses, but I see that's not an exported symbol.

[hanshuebner]

CL-USER> (let ((s (flex:make-in-memory-output-stream)))
           (write-byte 1 s)
           (write-byte 2 s)
           (flex:get-output-stream-sequence s))
#(1 2)

What's missing?

[Zulu-Inuoe]

I don't want to create a new sequence. I want to write to an existing string with a fill pointer

For example, I can

(let ((string (make-array 0 :element-type 'character :adjustable t :fill-pointer t)))
  (with-output-to-string (stream string)
    (write-line "Hello, world!" stream))
  string)  #| => "Hello, world!" |#

but there is no way for me to keep that stream alive as with-output-to-string closes it on lexical exit

[hanshuebner]

Ah, understood. I guess that you'll need to come up with a pull request to support that.

[Zulu-Inuoe]

Gotcha - I wanted to make sure such a thing didn't already exist without me knowing.

I ended up doing the following:

(defclass %string-output-stream (trivial-gray-streams:fundamental-character-output-stream)
  ((%string :initarg :string)))

(defmethod trivial-gray-streams:stream-write-char ((stream %string-output-stream) character)
  (vector-push-extend character (slot-value stream '%string)))

(defmethod trivial-gray-streams:stream-write-string ((stream %string-output-stream) string &optional start end)
  (let* ((%string (slot-value stream '%string))
         (len (- (or end (length string)) (or start 0)))
         (prev-len (fill-pointer %string))
         (available-space (- (array-dimension %string 0) prev-len))
         (new-len (+ prev-len len)))
    (when (< available-space len)
      (adjust-array %string new-len))
    (setf (fill-pointer %string) new-len)
    (replace %string string :start1 prev-len :start2 (or start 0) :end2 end)))

which seems to work, but haven't confirmed if there's some missing part of the protocol I'm unaware about. I'll try and put some effort into that and open a PR.

Thanks

[avodonosov]

@Zulu-Inuoe, the standard function is MAKE-STRING-OUTPUT-STREAM

http://www.lispworks.com/documentation/lw50/CLHS/Body/f_mk_s_2.htm

[Zulu-Inuoe]

@avodonosov That does not allow me to write to an existing string

[avodonosov]

@Zulu-Inuoe, ah, I see.

Then it's even a kind of a defect in the standard, IMHO, beoause with-output-to-string can not be trivially implemented it terms of make-string-output-stream.

[Zulu-Inuoe]

It actually can, by

(defmacro my/with-output-to-string ((var str) &body body)
  `(let ((,var (make-string-output-stream)))
    ,@body
    (my/append-chars-to ,str (get-output-stream-string ,var))))

Which is, of course, inefficient, but has the same general behaviour (with the exception that it'll error "late" if the destination string is too small).

But that's neither here nor there .. I ended up just writing my own class + specializations as noted above, but I still think this is a reasonable use-case if somebody is inspired to add to flexi-streams. Like I said, it works for me but I don't know if there's more to the protocol eg stream-element-type, wirte-char vs write-sequence, file-position, and so on

[avodonosov]

For the protocol, see the original Gray proposal: http://www.nhplace.com/kent/CL/Issues/stream-definition-by-user.html, check at least the "Character output:" section.

While most of the generic functions there are specified to have default methods, current CMUCL misses stream-line-column default method (https://github.com/trivial-gray-streams/trivial-gray-streams/issues/12), so provide yours, even if just returning NIL

But simply returning NIL will deprive FORMAT ~T and FRESH-LINE of current column information, so they will work in the most dumb way. You may want to provide a real implementation of the stream-line-column.

Read the proposal for what else may be required, I am not used to implementing custom streams.

In addition to the original proposal, consider 3 methods which are missing there stream-write-sequence , stream-read-sequence, stream-file-position. Currently trivial-gray-streams does not specify whether default methods exist or user has to provide his own. Trivial-gray-streams simply inherits this decision from the lisp implementation. Most implementations, I think, provide default methods. For example, write-sequence by default will usually result in repeated calls to stream-write-char, probably. But it's better to implement stream-write-sequence yourself, to avoid unexpected failures on lisp implementations other than yours. As for stream-file-position do you wan to support it? Then implement. Since you seem to care about output only, stream-read-sequence is not needed. (Although, why not implement input / output string stream?)

In general, you may first write a test with as many CL output functions as possible

    (let ((stream (make-my-custom-string-stream ...)))
        (write-char s  ... )
        (write-string s  ... )
        (write-sequence s  ... )
        (format s "~T..." ... )
        (pprint s ... )

       )

and see if the results satisfy you.

After all, your new class does not modify any existing behavior, it's purely additional feature, so it can be released in whatever form it is, and then fixed later, if any omissions are discovered. I am not a flexi-streams maintainer and not sure such a feature belongs to the flexi-streams scope; but that can also be a separate little library.