mirror of
https://gitlab.com/AutumnMeowMeow/jexer
synced 2024-09-19 11:50:19 -06:00
simplified proposal
This commit is contained in:
parent
9dbe0d218e
commit
9a1dfe2bd9
1 changed files with 495 additions and 0 deletions
495
docs/images2.md
Normal file
495
docs/images2.md
Normal file
|
@ -0,0 +1,495 @@
|
|||
Terminal Emulator Images Standard - Proposed Design - Simplified
|
||||
================================================================
|
||||
|
||||
Version: 1
|
||||
|
||||
|
||||
|
||||
Purpose
|
||||
-------
|
||||
|
||||
See the [original proposal](images.md) for purpose, design goals, and
|
||||
definitions.
|
||||
|
||||
This document is an updated proposal to address feedback on the first
|
||||
proposal, which included: "overengineered", "hopelessly
|
||||
overengineered", and "unnecessarily complex."
|
||||
|
||||
I perceive this feedback as a positive: it is far easier to imagine a
|
||||
feature and remove it, than to fail to picture it and need it later.
|
||||
The original proposal was a superset of every image format referenced,
|
||||
and generalized beyond to multimedia. This proposal is sharply
|
||||
reduced from that to: "put this pixel rectangle from the image, into
|
||||
that cell-based rectangle with specific scaling policy". It is mostly
|
||||
a subset of the iTerm2 protocol, with specifications for what happens
|
||||
to the cursor, and more precise definitions of the
|
||||
"preserveAspectRatio" equivalent options.
|
||||
|
||||
|
||||
|
||||
Tradeoffs
|
||||
---------
|
||||
|
||||
Simplifying the original proposal will significantly reduce
|
||||
complexity, but also eliminates features. The major tradeoffs offered
|
||||
in this revised proposal are:
|
||||
|
||||
1. Elimination of the layers feature, and with it the ability to place
|
||||
images behind text. In this proposal, a Cell on the screen will
|
||||
show either a (part of a) visible image, or a (part of a) text
|
||||
glyph, but never both.
|
||||
|
||||
2. Elimination of the "url" option, and with it the ability for an
|
||||
application to specify a filename or other method for the terminal
|
||||
to find the file data on the local machine. Image data must always
|
||||
be passed inline with the sequences.
|
||||
|
||||
3. Elimination of response codes, and with it:
|
||||
|
||||
- The ability for multiplexers to blindly pass on the sequences to
|
||||
their host terminal.
|
||||
|
||||
- The ability for applications to reliably detect success or
|
||||
failure of image display operations.
|
||||
|
||||
4. Elimination of pixel-oriented image placement operations, and with
|
||||
it the ability of applications to pass on image calculations to the
|
||||
terminal. An application which requires pixel-perfect rendering
|
||||
must generate the pixels it needs, aligned such to be displayed at
|
||||
the top-left corner of the text Cell rectangle.
|
||||
|
||||
|
||||
|
||||
Summary
|
||||
-------
|
||||
|
||||
This revised document proposes two independent new features:
|
||||
|
||||
1. A method to transfer image data for immediate display within the
|
||||
screen Cell grid ("Direct Images").
|
||||
|
||||
2. A method to transfer image data to a terminal-managed cache, and
|
||||
later display that data within the screen Cell grid ("Cached
|
||||
Images").
|
||||
|
||||
The only difference between the first and second feature is the
|
||||
presence of an ID key. Direct images do not use an ID key, while
|
||||
cached images use a store operation with ID key followed by one or
|
||||
more display operations with ID key.
|
||||
|
||||
Images are applied to text Cells, and once set handled the same way
|
||||
text Cells are handled: erasing a line erases the image Cells on that
|
||||
line, inserting a character will shift image Cells on that row over,
|
||||
scrolling will shift the image up, and so on. Therefore, terminals
|
||||
will need to be prepared for the scenario that every Cell on the
|
||||
display is a separate image, with a separate display scaling option
|
||||
that will need to be re-applied automatically if font metrics change.
|
||||
|
||||
|
||||
|
||||
All Features - Detection
|
||||
------------------------
|
||||
|
||||
Applications can detect support for these features using Primary
|
||||
Device Attributes (DA) and DECID (ESC Z, or 0x9A).
|
||||
|
||||
Terminals that support this standard will repond with additional
|
||||
parameter(s): "224" for direct images and "225" for cached images. A
|
||||
recap of the parameters xterm supports is listed below, with these new
|
||||
feature responses included:
|
||||
|
||||
| VT220 (and higher) Response | Description |
|
||||
|-----------------------------|--------------------------------------------|
|
||||
| 1 | 132-columns |
|
||||
| 2 | Printer |
|
||||
| 3 | ReGIS graphics |
|
||||
| 4 | Sixel graphics |
|
||||
| 6 | Selective erase |
|
||||
| 8 | User-defined keys |
|
||||
| 9 | National Replacement Character sets |
|
||||
| 1 5 | Technical characters |
|
||||
| 1 6 | Locator port |
|
||||
| 1 7 | Terminal state interrogation |
|
||||
| 1 8 | User windows |
|
||||
| 2 1 | Horizontal scrolling |
|
||||
| 2 2 | ANSI color, e.g., VT525 |
|
||||
| 2 8 | Rectangular editing |
|
||||
| 2 9 | ANSI text locator (i.e., DEC Locator mode) |
|
||||
| 2 2 4 | Direct Images Version 1 |
|
||||
| 2 2 5 | Cached Images Version 1 |
|
||||
|
||||
|
||||
|
||||
Direct Images - Summary
|
||||
-----------------------
|
||||
|
||||
Non-text data (images) can be sent to the terminal for immediate
|
||||
display in a rectangular region of text Cells. Image data is
|
||||
transmitted to the terminal using a wire format described later in
|
||||
this document.
|
||||
|
||||
Setting a Cell to image is a destructive operation: the Cell's
|
||||
original text is lost. Similarly, setting a Cell (or multiple Cells
|
||||
for fullwidth glyphs or grapheme clusters) to text is a destructive
|
||||
operation: the image in the Cell(s) is lost.
|
||||
|
||||
Setting any part of a multi-Cell Tile to image also "breaks up" the
|
||||
Tile into a range of single Cells. In other words, image data can
|
||||
only be carried by a Cell, not a Tile.
|
||||
|
||||
|
||||
|
||||
Direct Images - New Sequences
|
||||
-----------------------------
|
||||
|
||||
A terminal with direct images feature must support the following new
|
||||
sequences:
|
||||
|
||||
| Sequence | Description |
|
||||
|--------------------------------------|-------------------------|
|
||||
| OSC 1 3 3 8 ; F i l e = {args} : {data} BEL | Display image at (x, y) |
|
||||
| OSC 1 3 3 8 ; F i l e = {args} : {data} ST | Display image at (x, y) |
|
||||
|
||||
|
||||
|
||||
For the OSC 1 3 3 8 sequence:
|
||||
|
||||
* The {args} is a set of key-value pairs (each pair separated by
|
||||
semicolon (';')), followed by a colon (':'), followed by a base-64
|
||||
encoded string ({data}).
|
||||
|
||||
* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
|
||||
'a' - 'z').
|
||||
|
||||
* A value is any printable ASCII string not containing whitespace,
|
||||
colon, or semicolon ('!' - '9', '<' - '~').
|
||||
|
||||
* Any alpha-numeric key may be specified. A key that is not supported
|
||||
by the terminal is ignored without error.
|
||||
|
||||
* The image is processed as shown below:
|
||||
|
||||
- The pixels are drawn starting at the upper-left corner of the text
|
||||
cursor position.
|
||||
|
||||
- If scroll is specified as 1 (enabled), then:
|
||||
|
||||
a. The screen is scrolled up if the image overflows into the
|
||||
bottom text row.
|
||||
|
||||
b. The cursor's final position is on the same column as the
|
||||
starting cursor position, and on the row immediately below the
|
||||
image.
|
||||
|
||||
- If scroll is omitted or specified as 0 (disabled), then:
|
||||
|
||||
a. The screen is never scrolled.
|
||||
|
||||
b. Pixels that would be drawn below the visible region on screen
|
||||
are discarded.
|
||||
|
||||
c. The cursor's final position is at the same column and row as
|
||||
the starting cursor position, i.e. the cursor does not move at
|
||||
all.
|
||||
|
||||
- Pixels that would be drawn to the right of the visible region on
|
||||
screen are discarded.
|
||||
|
||||
|
||||
|
||||
The keys for the key-value pairs that must be supported by the
|
||||
terminal are listed below:
|
||||
|
||||
| Key | Default Value | Description |
|
||||
|--------------|---------------|----------------------------------------------|
|
||||
| type | "image/rgb" | mime-type describing data field |
|
||||
| width | 1 | Number of Cells or pixels wide to display in |
|
||||
| height | 1 | Number of Cells or pixels high to display in |
|
||||
| scale | "none" | Scale/zoom option, see below |
|
||||
| sourceX | 0 | Media source X position to display |
|
||||
| sourceY | 0 | Media source Y position to display |
|
||||
| sourceWidth | "auto" | Media width in pixels to display |
|
||||
| sourceHeight | "auto" | Media height in pixels to display |
|
||||
| scroll | 0 | If 0, scroll the display if needed |
|
||||
|
||||
A terminal may support additional keys. If a key is specified but not
|
||||
supported by the terminal, then it is ignored without error.
|
||||
|
||||
|
||||
|
||||
The "type" value is a mime-type string describing the format of the
|
||||
base64-encoded binary data. The terminal must support at minimum these
|
||||
mime-types:
|
||||
|
||||
| Type String | Description |
|
||||
|---------------|--------------------------------------------------------------|
|
||||
| "image/rgb" | Big-endian-encoded 24-bit red, green, blue values |
|
||||
| "image/rgba" | Big-endian-encoded 32-bit red, green, blue, alpha values |
|
||||
| "image/png" | PNG file data as described by (reference to PNG format) |
|
||||
|
||||
A terminal may support additional types. An application can detect
|
||||
terminal support for a format by:
|
||||
|
||||
1. Attempt to draw image, with "scroll" set to 1.
|
||||
|
||||
2. Check cursor position DSR 6.
|
||||
|
||||
3. If cursor has moved, then the terminal supports this image type.
|
||||
|
||||
|
||||
|
||||
The "width" and "height" values are positive integers describing the
|
||||
number of Cells the image will be placed in.
|
||||
|
||||
|
||||
|
||||
The "scale" value can take the following values:
|
||||
|
||||
| Value | Meaning |
|
||||
|------------|---------------------------------------------------------------|
|
||||
| "none" | No scaling along either axis. |
|
||||
| "scale" | Stretch image, preserving aspect ratio, to maximum size in the target area without cropping |
|
||||
| "stretch" | Stretch along both axes, distorting aspect ratio, to fill the target area |
|
||||
| "crop" | Stretch along both axes, preserving aspect ration, to completely fill the target area, cropping pixels that will not fit |
|
||||
|
||||
|
||||
|
||||
"sourceX", "sourceY", "sourceWidth", and "sourceHeight" define the
|
||||
rectangle of pixels from the media that will be displayed on the
|
||||
screen. The ranges for these values is shown below:
|
||||
|
||||
| Key | Minimum Value | Maximum Value | Default Value |
|
||||
|--------------|---------------|-------------------------------|---------------|
|
||||
| sourceX | 0 | Media's full width - 1 | 0 |
|
||||
| sourceY | 0 | Media's full height - 1 | 0 |
|
||||
| sourceWidth | 1 | Media's full width - sourceX | "auto" |
|
||||
| sourceHeight | 1 | Media's full height - sourceY | "auto" |
|
||||
|
||||
If any of these values are specified and outside the range, no image
|
||||
is displayed, and the cursor does not move. "sourceWidth" and
|
||||
"sourceHeight" can be "auto", which means use the maximum available
|
||||
width/height (given sourceX/sourceY) from the media's inherent
|
||||
dimensions.
|
||||
|
||||
|
||||
|
||||
Cached Images - Summary
|
||||
-----------------------
|
||||
|
||||
Non-text data (image) can be sent to the terminal for later display in
|
||||
a rectangular region of text Cells. Image data is transmitted to the
|
||||
terminal using the CSTORE command described below, and displayed on
|
||||
screen using the CDISPLAY command. A single CSTORE command can
|
||||
support many CDISPLAY commands.
|
||||
|
||||
Upon display, setting a Cell to image is a destructive operation: the
|
||||
Cell's original text is lost. Similarly, setting a Cell (or multiple
|
||||
Cells for fullwidth glyphs or grapheme clusters) to text is a
|
||||
destructive operation: the image in the Cell(s) is lost.
|
||||
|
||||
Setting any part of a multi-Cell Tile to image also "breaks up" the
|
||||
Tile into a range of single Cells. In other words, image data can
|
||||
only be carried by a Cell, not a Tile.
|
||||
|
||||
|
||||
|
||||
Cached Images - Cache/Memory Management
|
||||
---------------------------------------
|
||||
|
||||
The terminal manages a cache of multimedia data on behalf of the
|
||||
application. The application requests media be stored in the cache
|
||||
and provides an ID. This ID is later used to request display on the
|
||||
screen.
|
||||
|
||||
The amount of memory and retention/eviction strategy for the cache is
|
||||
wholly managed by the terminal, with the following restrictions:
|
||||
|
||||
* The terminal may not remove items from the cache that have any
|
||||
portion being actively displayed on the primary or alternate
|
||||
screens.
|
||||
|
||||
The scrollback buffer is permitted, and recommended, to contain only a
|
||||
few (or zero) multimedia images. Terminals should consider retaining
|
||||
only the last 2-5 screens' worth of pixel data in the scrollback
|
||||
buffer.
|
||||
|
||||
Applications have no control over when images are removed from the
|
||||
cache, and no provision is made to generate/ensure unique IDs.
|
||||
|
||||
A terminal multiplexer that passes all CSTORE/CDISPLAY commands to the
|
||||
host terminal will need to parse the CSTORE and CDISPLAY sequences for
|
||||
the "id" field and rewrite it to be unique for all of its inner
|
||||
terminals.
|
||||
|
||||
|
||||
|
||||
Cached Images - New Sequences
|
||||
-----------------------------
|
||||
|
||||
A terminal with cached images feature must support the following new
|
||||
sequences:
|
||||
|
||||
| Sequence | Command | Description |
|
||||
|--------------------------------------|-----------|-------------------------|
|
||||
| OSC 1 3 4 0 ; F i l e = {args} : {data} BEL | CSTORE | Display media at (x, y) |
|
||||
| OSC 1 3 4 1 ; Pi ; {args} ST | CDISPLAY | Display media at (x, y) |
|
||||
|
||||
|
||||
|
||||
Cached Images - CSTORE
|
||||
----------------------
|
||||
|
||||
For the CSTORE command:
|
||||
|
||||
* The {args} is a set of key-value pairs (each pair separated by
|
||||
semicolon (';')), followed by a colon (':'), followed by a base-64
|
||||
encoded string ({data}).
|
||||
|
||||
* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
|
||||
'a' - 'z').
|
||||
|
||||
* A value is any printable ASCII string not containing whitespace,
|
||||
colon, or semicolon ('!' - '9', '<' - '~').
|
||||
|
||||
|
||||
|
||||
The keys for the key-value pairs that must be supported by the
|
||||
terminal are listed below:
|
||||
|
||||
| Key | Default Value | Description |
|
||||
|--------------|---------------|----------------------------------------------|
|
||||
| id | 0 | ID to refer to the image |
|
||||
| type | "image/rgb" | mime-type describing data field |
|
||||
|
||||
|
||||
|
||||
The "id" value is a non-negative integer between 0 and 999999.
|
||||
|
||||
|
||||
|
||||
The "type" value is a mime-type string describing the format of the
|
||||
base64-encoded binary data. The terminal must support at mimunum these
|
||||
mime-types:
|
||||
|
||||
| Type String | Description |
|
||||
|---------------|--------------------------------------------------------------|
|
||||
| "image/rgb" | Big-endian-encoded 24-bit red, green, blue values |
|
||||
| "image/rgba" | Big-endian-encoded 32-bit red, green, blue, alpha values |
|
||||
| "image/png" | PNG file data as described by (reference to PNG format) |
|
||||
|
||||
A terminal may support additional types. An application can detect
|
||||
terminal support for a format by:
|
||||
|
||||
1. Store image in cache.
|
||||
|
||||
2. Attempt to draw image, with "scroll" set to 1.
|
||||
|
||||
3. Check cursor position DSR 6.
|
||||
|
||||
4. If cursor has moved, then the terminal supports this image type.
|
||||
|
||||
|
||||
|
||||
Cached Images - CDISPLAY
|
||||
------------------------
|
||||
|
||||
For the CDISPLAY command:
|
||||
|
||||
* Pi - a non-negative integer ID that was used in a previous CSTORE
|
||||
command.
|
||||
|
||||
* The {args} is a set of key-value pairs (each pair separated by
|
||||
semicolon (';')), followed by a colon (':'), followed by a base-64
|
||||
encoded string.
|
||||
|
||||
* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
|
||||
'a' - 'z').
|
||||
|
||||
* A value is any printable ASCII string not containing whitespace,
|
||||
colon, or semicolon ('!' - '9', '<' - '~').
|
||||
|
||||
* Any alpha-numeric key may be specified. A key that is not supported
|
||||
by the terminal is ignored without error.
|
||||
|
||||
* The image pixels are processed as shown below.
|
||||
|
||||
- The pixel are drawn starting at the upper-left corner of the text
|
||||
cursor position.
|
||||
|
||||
- If scroll is specified as 1 (enabled), then:
|
||||
|
||||
a. The screen is scrolled up if the image overflows into the
|
||||
bottom text row.
|
||||
|
||||
b. The cursor's final position is on the same column as the
|
||||
starting cursor position, and on the row immediately below the
|
||||
image.
|
||||
|
||||
- If scroll is omitted or specified as 0 (disabled), then:
|
||||
|
||||
a. The screen is never scrolled.
|
||||
|
||||
b. Pixels that would be drawn below the visible region on screen
|
||||
are discarded.
|
||||
|
||||
c. The cursor's final position is at the same column and row as
|
||||
the starting cursor position, i.e. the cursor does not move at
|
||||
all.
|
||||
|
||||
- Pixels that would be drawn to the right of the visible region on
|
||||
screen are discarded.
|
||||
|
||||
|
||||
|
||||
The keys for the key-value pairs that must be supported by the
|
||||
terminal are listed below:
|
||||
|
||||
| Key | Default Value | Description |
|
||||
|--------------|---------------|----------------------------------------------|
|
||||
| id | 0 | ID to refer to the image |
|
||||
| width | 1 | Number of Cells or pixels wide to display in |
|
||||
| height | 1 | Number of Cells or pixels high to display in |
|
||||
| scale | "none" | Scale/zoom option, see below |
|
||||
| sourceX | 0 | Media source X position to display |
|
||||
| sourceY | 0 | Media source Y position to display |
|
||||
| sourceWidth | "auto" | Media width in pixels to display |
|
||||
| sourceHeight | "auto" | Media height in pixels to display |
|
||||
| scroll | 0 | If 1, scroll the display if needed |
|
||||
|
||||
A terminal may support additional keys. If a key is specified but not
|
||||
supported by the terminal, then it is ignored without error.
|
||||
|
||||
|
||||
|
||||
The "width" and "height" values are positive integers describing the
|
||||
number of Cells the image will be placed in.
|
||||
|
||||
|
||||
|
||||
The "scale" value can take the following values:
|
||||
|
||||
| Value | Meaning |
|
||||
|------------|---------------------------------------------------------------|
|
||||
| "none" | No scaling along either axis. |
|
||||
| "scale" | Stretch image, preserving aspect ratio, to maximum size in the target area without cropping |
|
||||
| "stretch" | Stretch along both axes, distorting aspect ratio, to fill the target area |
|
||||
| "crop" | Stretch along both axes, preserving aspect ration, to completely fill the target area, cropping pixels that will not fit |
|
||||
|
||||
|
||||
|
||||
"sourceX", "sourceY", "sourceWidth", and "sourceHeight" define the
|
||||
rectangle of pixels from the media that will be displayed on the
|
||||
screen. The ranges for these values is shown below:
|
||||
|
||||
| Key | Minimum Value | Maximum Value | Default Value |
|
||||
|--------------|---------------|-------------------------------|---------------|
|
||||
| sourceX | 0 | Media's full width - 1 | 0 |
|
||||
| sourceY | 0 | Media's full height - 1 | 0 |
|
||||
| sourceWidth | 1 | Media's full width - sourceX | "auto" |
|
||||
| sourceHeight | 1 | Media's full height - sourceY | "auto" |
|
||||
|
||||
If any of these values are specified and outside the range, no image
|
||||
is displayed, and the cursor does not move. "sourceWidth" and
|
||||
"sourceHeight" can be "auto", which means use the maximum available
|
||||
width/height (given sourceX/sourceY) from the media's inherent
|
||||
dimensions.
|
Loading…
Reference in a new issue