[Lispweb] Regarding: AServe & :FORMAT :BINARY, Streams, Sockets, DocBook, furthermore
Sean Champ
schamp at commonwerx.org
Fri Sep 9 08:29:44 CDT 2005
Hello,
This is a long email. Obviously, not everyone will have time to read
it.
* Summaries
Some Products addressed, herein -- references mentioned in "The
Summaries" below:
- AServe
- CL-XML (briefly)
- Swank (from SLIME)
- DocBook (footnote [2])
- IMHO (tangentially, but about the bufferng stram reader, in IMHO;
see also footnote [6], for the original code of the 1.3.2 release
of IMHO, though I have no idea how it will be available, in any more
-- whether OnShore will release IMHO again, and how and when my
IMHO-Tioga fork upon IMHO would be made available)
Additioanl notes:
1) Emacs [footnote] support is pretty bare. As a consequence,
some footnote numbers have become jumbled, here. My apologies.
2) Secondly: I'm used to writing with DocBook. I'm used to writing
long things of text, in DocBook. Conceivably, HTML would be even
easier than this kludgy text/plain, to organize such within --
but, HTML is not supported so well, in maybe-all email
clients, and HTML emails are not such that a mailing lists will
tend to go well with. So, here is the long text plain. I will not
conceive of apologizing about it. If this will be of interest,
that is why I have written this. If it will not be of interest, I
will have to adapt some code for AServe, myself, with zero
'input'; it would be started, then, in no time soon.
The summaries:
-> In the body of this: AServe, and the means for making a
DO-HTTP-REQUEST to return a stream. The meat of the proposal starts
at "* Actually", below.
-> Footnote [3]: Regarding a portable socket library project
(one proposed name: Socket-Kit), and the code and develoeprs of
Swank, regarding such code that the socket kit would be derived upon,
of the SLIME that is the project containing of Swank.
-> Footnote [1]: The most of why the patches mentioned about CL-XML
are not available.
-> Footnote [2]: Mentioning a CommonBook project, as a fork
proposed upon DocBook, not to be singularly relevant about the
documentaiton of Common Lisp code, but presumably, with a whole
module to be made about Common Lisp.
-> Footnote [2]: Also mentioning what is the hitch about why this has
not been to the occurracne of any more code, of any more work, of
any more sign of my going to-do list. BUDGET is the matter. This
list is not a Common Lisp Chamber of Commerce, granted. Yet, this
matter of concern about budget, in the programming of Common Lisp
software, it will be resolved, or any of this would be made -- if
made of my person -- in the utmost menial trickle, in little
mention of anything, and probably with such long emails as this. I
am at dire ends to resolve my budget. While it is a desperate
maneuver, to say it here, and I will not pretend as if it was not,
this will be the first and the last I will mention, of it, on this
list.
-> Foonote [7]: SLIME's Swank also supports cross-implementation
thread operation. It may be of use as a reference, about such.
-> Footnote [8]: Not my first prirority, to separate the buffering
stream support from IMHO (1.3.2), and provide it separately, for
implementaiton with any Lisp's gray-stream forms, but it can be
done.
-> Footnote [6]: Referencing IMHO, in its publicly available form.
* Towards the point
I had adapted some of the code of CL-XML, for to use AServe's
DO-HTTP-REQUEST, for CL-XML's WITH-HTTP-STREAM.[1] After I had done
so, I had forgotten that DO-HTTP-REQUEST -- when given no :FORMAT
argument, and a certain varaible would not have :BINARY as its value
-- then DO-HTTP-REQUEST will return a string.
I had had to make a stream out of the string it would return. Then, I
had forgotten that I had done so, and had begun to think as if
DO-HTTP-REQUEST would return a stream. I think it should; that is the
main matter of this email.
I had written this email, then, upon the incorrect recollection, as if
it was as I preferred it to be, as if a stream would be returned by
DO-HTTP-REQUEST, and it is not. DO-HTTP-REQUEST returns either (1) a
string or (2) such as a (SIMPLE-ARRAY (UNSIGNED-BYTE 8))
I will not put so much time into this email, as try to revise this
in all, upon my recollection, now, that it is a *vector* returned by
DO-HTTP-REQUEST. So, if I have incorrectly stated the matter, below,
there has been my explanation.
* The Proposal: DO-HTTP-REQUEST* => STREAM
I suppose that I should begin this, with the proposal -- regarding
Aserve -- that something *like* DO-HTTP-REQUEST would be defined, such
that it would "do the same thing", for HTTP request/response handling,
but would return a *stream*.
Excluding the footnotes, the rest of this email is about: "What type
of a stream, and how to make it?"
** Proposed Name for New Function: DO-HTTP-REQUEST*
A proposal, about naming: The proposed stream-returning form might
be named DO-HTTP-REQUEST*, so that it will not break any code that
would already rely upon DO-HTTP-REQUEST returning such values as it
returns.
* Second Introduction, or see: "* Actually, The Proposals", below
The primary matter of this email, originally: That DO-HTTP-REQUEST,
when given a :FORMAT :BINARY, will return an object like of the type
(SIMPLE-ARRAY (UNSIGNED-BYTE 8))
I need it to return a stream (but as it had been, I needed it to
return a plain string, and then I had forgotten that that is how
it works -- DO-HTTP-REQUEST returning a vector, regardless of the
array-element-type of the vector it would return.)
** Also Binary Streams
In an additional note: It would sometimes need to be a "binary stream",
not always a "character stream", returned by DO-HTTP-REQUEST*
** Related Mater: {Remote, Local} Charcter Encodings
I am not sure what to say of a related matter, but it seems to bear
a note, presumably in the AServe documentation: About how the
establishment of the Lisp character stream may/will effect any changes
in the character encodings of the original (remote, HTTP retrieved)
document.
It seems that this matter of remote->local character-set
transformation should be addressed about the return value of the
standard DO-HTTP-REQUEST, furthermore, as about the element type of
the string/vector it would return.
I would be willing to address such, in the AServe documentation,
myself, but I should admit that it would be (really) hard to address
if without using DocBook, for the documentation of the matter. DocBook
is not used in the AServe documentation, however; it seems that the
Portable AServe documentation is provided as it was made of Franz Inc.[2]
** CONTENT-TYPE - Character Set - Relevance for String Stream
In addition: When the CONTENT-TYPE header of an HTTP response would
contain the string "; charaset=" then, the element type of a string
stream may be determined, upon that CONTENT-TYPE header. The code for
AServe may have to be patched for this. (I wanted to mark a note
about it, here and now.)
** OK
Tangents aside, and my implict desperation about budget, aside, this
matter of proposal -- about stream support in AServe's HTTP client
operations, and about some matters appearing as related, and then
... etc -- it is continued, below.
* Actually, The proposals
I have tried to address, here, two proposals about how this *may* be
done -- how a stream reader may be made for a DO-HTTP-REQUEST with
:FORMAT :BINARY
The proposals, as I see it, include:
1) * Socket Stream * : Just use the the stream for the socket, as
it would be available within the implementation's socket
implementaiton (and if necessary, use SLIME's Swank, as a
reference for porting this[3]). This would be suitable, for when the
stream would need to be read from, exactly once.
2) * Buffering Stream * : Make a buffer stream, containing such
information that would be read from the socket's stream.
Of both of those types of stream, either one could be useful, as the
return value from (DO-HTTP-REQUEST URL :FORMAT :BINARY)
I would propose that both types of streams would be made available.
I should even be able to implement support for both. However, I would
not know how the design of AServe's DO-HTTP-REQUEST would be best
resolved for it.
It would take more than the :FORMAT argument. The FORMAT value would
be used to determine the format of the elements in the stream. :FORMAT
would not be used to determine whether the stream would be a buffered
stream or a a one-pass-only (generally, when made directly on the
socket) stream. So, for DO-HTTP-REQUEST, a :BUFFER-P argument
... might be suggested.
* The introduction
Given the following, executed in SBCL:
(use-package :net.aserve.client)
(defparameter *v*
(do-http-request "http://google.com/" :format :binary))
(type-of *v*)
=> (SIMPLE-ARRAY (UNSIGNED-BYTE 8) (2390))
Of course, a SIMPLE-ARRAY cannot be READ from, directly, as if it was
like a STREAM.
* General Proposals, Sketches, Etc.
I know that some gray streams[5] support would be workable, upon the
array returned by (DO-HTTP-REQUEST URI :FORMAT :BINARY). Yet, I am
afraid if that would not be the most efficient method possible, for
making a stream upon the server's response.
I hope that there would be a means for working-up a stream that would
read directly from the socket. Yet, I am not so familiar with socket
and stream operations, not enough to determine, immediately, how this
would be done -- how a stream would be defined, such that the stream
would be read directly from the lisp implementation's socket. Granted,
I can look into the code of SLIME's Swank, for this.
I persume that such on-socket stream, if used within AServes
client.cl code, it may be made to operate within the method
(CLIENT-REQUEST-READ-SEQUENCE T CLIENT-REQUEST)
...in package :net.aserve.client
So, however the stream will be defined, Swank will be a part of
research for it.
If that CLIENT-REQUEST-READ-SEQUENCE method would not be the right
place to address the forms for it, I would be glad to hear a
suggestion about where would be the best place.
The question might remain, however: What sort of a stream to make?
Presumably, the socket's stream would not be such that may be used
with both forms:
(READ STREAM)
(FILE-POSITION STREAM :START))
For a one-pass reader operation, such that the reader would not have
to read the stream again, then the socket's stream would be
enough.
Yet, if anyone would need to have a socket's stream, such that would
be read over, then the cursor/point/file-position backed-up, then the
stream read over again, then there may be a want for such a buffered
stream reader as that which is addressed, below.
I had written the following, presuming it would be of use. Granted, it
may be not directly pertinent about this matter of :FORMAT :BINARY
* Buffering Stream Reader ?
(a proposal, regarding AServe, about a possible source of a buffered
stream reader -- such a reader that might be applied upon the socket
of a server's response to an HTTP request -- if this would be the only
best means for it)
As memory serves, IMHO 1.3.2 (see [6]) had included some support for
something like a buffering stream reader/writer. It uses arrays, in
the operations of it.
Though OnShore Development had ceased to provide recent versions of
it, yet IMHO is and does remains as an open-source product. Marco
Baringer has even ensured that IMHO would remain available; that is
what the note in [6] is regarding.
To continue it, in development, I had forked the 1.3.2 version of
IMHO, then calling it IMHO-Tioga. I have begun some work upon it,
though it has been unupublished. It not been much -- just renaming
some things in the C and the Lisp of IMHO, in preparation for support
of a newer mod_webapp, and getting a start at making IMHO work with
SBCL.
So, regardless of more projects:
If there would be no better way for to establish a Lisp (gray[5])
stream upon a socket, then IMHO's implementation of a buffering stream
reader/writer (whether from in IMHO or IMHO-Tioga, or however) but
*the* buffering-stream forms may be put into a separate package.[8] I
would be willing to do such, after I would make such forms to be
applicable upon AServe's client operations.
There is an issue, however, at that I am not sure if that is the best
means for addressing this matter of :FORMAT :BINARY.
I would prefer to simply have (if but to make, though I am not yet
sure of how best to begin about it, without affecting anything in
regards to character encodings) a stream reading directly from the
socket, with no arrays in between -- presuambly, leaving it to the
operating system, to do the buffering, in how the OS would handle
the socket object.
If it would be so much as a raw, binary stream, then it should be
enough for working-in with the XML parser in CL-XML. I just need some
sort of a stream, to read from; once that would be done, I can
continue with the patching and with the patched code.
* END
This email is long enough. The reader may pardon, if I have not edited
it through, enough.
To mention, to be sure: CL-XML, I intend to apply for more of
free/open source products. An RSS aggregator with support like for
Podcasts, an XBEL editor (with Garnet, frankly), Dublin Core fun, etc
-- it's some medium-distance stuff. Not all of it will require a
stream from DO-HTTP-REQUEST, but some of it, undoubtedly, will -- as
for support of UTF-8 encoded documents, however such would have to be
done, as per each Lisp implementation, and the stream upon the HTTP
response body --- and then any generation for Podcatss, plainer RSS
feeds, etc, presumably in IMHO-TIoga, possibly (ultimately, after such
woud be defined, if not firstly "just in IMHO-Tioga") upon a more
general, cross-application specification for types and operative
interfaces, for HTTP requests and responses (at least prototyped in
IMHO-Tioga and AServe, as I'd propose it)
An IMHO server could even work, in the same image, with an AServe
server, for establishing an HTTP based monitor upon the IMHO
server. That, then, would be another to-do item. If I can but get
onwards with XML, then hopefully it will be more organized than this.
A Lisp XML working group, I would hereby propose.
A liason from the group to XML.com, I would propose also. O'Reilly
Publishers might be contacted about it, conceivably.
Like, it might work, in Lisp:
(use-package :net.common-lisp) ; e.g
(defclass working-group (thing)
(...)
(:metaclass thing-class))
(defclass project (thing)
(...)
(:metaclass thing-class))
(make-instance 'working-group :name :XML)
(defuserintrface
(:something (project a-visitor-context html-server)
;; HTML generating forms go here, for project ... thing
)
(:something (project a-developer-context HTML-server)
;; wheeeeeeee
)
(:something (working-group a-admin-context HTML-server)
rubber baby buggie bumpers
(values 3/4)))
;; workflow diagrams could be generated upon such objects.
;; managers might even applaud, given something
;; more relevantly useful than bare interfaces
;; Yay, Lisp is a great language. It's easy to use, "doesn't always get
;; you hired" (?!) "but what the hey, times change."
Adios,
--
Sean Champ
schamp at commonwerx.org
----------------------------------------------------------------------
Footnotes:
[1] I have not yet been able to make the patches available, myself; I
have the hosting space for it, but no certainty about where it
will best be hosted within the hosting space. I've been in
communication with James Anderson, the developer of CL-XML. I
have submitted the patches to him, recently, and not bothered the
fellow, as for if or when the patches would be applied. This
matter of the AServe :FORMAT :BINARY -> array should be resolved,
before one of the patches would be applied, and the code of
CL-XML then distributed. I have simply not been sure of how best
to resolve it. It appears that some gray stream support for
AServe's :FORMAT :BINARY may be appropriate.
[2] ** CommonBook ** : DocBook is large, might seem somewhat
unwieldy, and maybe not quite modular enough. In its broadness,
however, DocBook is supportive of much of documentaiton, in SGML and
XML. Yet then, about Common Lisp code: DocBook has not been made
to be so well supportive, in its own, about Common Lisp forms
(e.g. DocBook's funcprototype element may esem to be particularly
relevant about C and Java). I've proposed, mutedly, that a
CommonBook project would be forked upon DocBook; to address such
in more would take a great amount of space, frankly. Should it be
of interest, however, perhaps there can be a project made, for
it. I do not mean to shun common-lisp.net, either, and neither to
be selfish about the project, but I would prefer to make it be
hosted upon my own host. If ever, anything about
project-management and component/product distribution might be hosted
there, also, in an effort that may be made in collaboration with
common-lisp.net, and hopefully, some matters about GForge and its
SOAP module. However, something of a crux of this, which I tend
to not mention, but possibly should: Budget remains a concern to
me, and I think it is rather more of a concern that most persons
reading this email would even have to realize. I will not make
this into a gross work, but budget has been the main issue of
concern, to me. I am not even employed as a programmer, and in
nothing whatsoever about software. Even if it is ironic, I have
attained to employment as a manual laborer, that being the best
that I have found, of my non-university-graduated self, literally
stuck in one Fresno California, with no transportation out of
here. Yet still, Fresno may be not always a bad place to be --
business-wise, geographically, and in some, even if some very
rare matters, culturally. I could use any earnest advice about,
how to resolve my concerns in regards to work and budget. I am
positive that my condition is not altogether singular, but may be
darned singular in its own -- picked up on Common Lisp, sometime
after abandoning, essentially, both theatre and college (not with
intent to have no education, but with no idea of what to procee
dabout, at the point of the abandonment) and then the student
loans became defaulted, and the hole was made deeper for it. My
naivete astounding -- even to me, now -- I will drag along in
grunt labor, if that is all there possibly is, and no software
will become, while I am put upon with it. It is "temporary" work,
may be taken to daily, if not weekly; it is low, but has been
enough to survive upon. This personal note ended, I would
appreciate any earnest adivce, towards resolving the matters here
expressed.
[3] ** a unified socket implementaiton ** might be extracted from
SLIME -- so to speak -- directly, from/upon Swank. Swank already
includes support for a number of socket implementations, across
Lisp implementations. Though one might find any qualms about
Swank's one-method-per-generic-function behaviors, as used for
it, yet I will not gripe about the design of it; it works. If I
would know a justification for it, I might accept it,
myself. If I would not, though, I would propose that only an
ordinary function would be used, after each defimplementaiton
call. Conceivably, then, Swank's defimplementaiton macro might
be modified, for this. The ultimate end would be that a most
efficient, portable socket library would be defined -- sort of
extracted from the SLIME project, and probably depended upon by
the SLIME project, as for the execution of Swank. If
common-lisp.net would pick up the proposal for this, I'd be
willing to carry through, about it, as a dveloper. In earnest, I
would propose that if anyone would not be bothered about my
comment, there, about that small matter of the design in Swank,
then such a project, possibly at common-lisp.net, might be
maintained by any of the Slime developers, probably of whom is
most in charge of Swank, if it would be accepted, the thought: A
portable socket library, essentially extracted from the SLIME
project (in seperation from Swank, though as such that Swank
would continue to use). This would be of use, then, for such as
the porting of any Common Lisp HTTP product, to make it work with as
many implementations as the portable socket kit would work
with. (Maybe Socket-Kit could be a name for the project; I like
the sound of the name, and how it might serve to make Common Lisp
seem "less heady, and very practical", as the Common Lisp
language is, even if anyone would not be aware of it, and even if
anyone would be wary of Common Lisp, if they would think it is a
"just a heady langauge". Not to wedge marketing in with this, but
the perception of the product is a part to the person's
decision in the adoption of the product, or the
non-adoption. Names, I know, may simply be names, as when they
are things that identify products, distinctly, as within a
programmatic system, regardless of what anyone would think of a
name. Neither am I so very latched on to the name Socket-Kit,
though I would propose it, as a name, for the portable socket
library. Regarding a Debian pacakge, cl-socket-kit could even be
fun to read.)
[4] Gray Streams are fully supported in SBCL, CMUCL, and probably
every other standard Common Lisp implementation. Simple Streams
have not been wholly supported in SBCL, and I am not sure about
CMUCL, there. Frankly, I'm just as happy to stick with gray
streams, and not concern myself about learning how to work with
simple streams.
[6] see http://www.cliki.net/IMHO and thanks to Marco Baringer, for
keeping the IMHO and ODCL code available.
[7] It may be of some additional interest -- albeit, not explicitly
about this matter -- but to draw it to attention: Swank supports
threads/processes, across Lisp implementations. The forms used
for such may be determined within the swank-<implementation>.lisp
files, e.g. swank-openmcl.lisp. Presumably, this would be of
interest, primarly, for developers intending to support threading,
across implementations. It might be of interest, for any more
general study -- at least, if not in most, of the interfaces
used by various implementations for Lisp -- for threading. I
meant to just mention this as a note (coincident with Swank's
cross implementation socket support) in case it would be of
interest. HERE AND BELOW, EMACS BROKE THE FOOTNOTES. (SORRY.)
[8] It would be a fine objective, to package the buffer stream forms,
seperate from IMHO / IMHO-Tioga. Yet, I will probably leave it
not done, until I would find a cause for it. Frankly, IMHO is not
the thing most in front of me. CL-XML and AServe's HTTP client
operations are, in how the latter would best be made to return a
stream.
More information about the lispweb
mailing list