mirror of
https://gitlab.com/AutumnMeowMeow/jexer
synced 2024-09-19 11:50:19 -06:00
524 lines
20 KiB
Markdown
524 lines
20 KiB
Markdown
Terminal Emulator Images Standard - Proposed Design - Simplified
|
|
================================================================
|
|
|
|
Version: 1
|
|
|
|
|
|
|
|
Purpose
|
|
-------
|
|
|
|
See the [original proposal](images.md) for purpose, design goals, and
|
|
definitions.
|
|
|
|
This document is an updated proposal to address feedback on the first
|
|
proposal, which included: "overengineered", "hopelessly
|
|
overengineered", and "unnecessarily complex." I perceive this
|
|
feedback as a positive: it is far easier to imagine a feature and
|
|
remove it, than to fail to picture it and need to shoehorn it in
|
|
later.
|
|
|
|
The original proposal was a superset of every image format referenced,
|
|
and generalized beyond to multimedia. This proposal is sharply
|
|
reduced from that to: "put this pixel rectangle from the image, into
|
|
that cell-based rectangle with specific scaling policy". It is mostly
|
|
a subset of the iTerm2 protocol, with:
|
|
|
|
* Specifications for what happens to the cursor.
|
|
|
|
* More precise definitions of the "preserveAspectRatio" equivalent
|
|
options.
|
|
|
|
* Explicit restriction to a Cell-based target region.
|
|
|
|
* Definition that pixels not covered by image are set to the current
|
|
background color.
|
|
|
|
|
|
|
|
Tradeoffs
|
|
---------
|
|
|
|
Simplifying the original proposal will significantly reduce
|
|
complexity, but also eliminates features. The major tradeoffs offered
|
|
in this revised proposal are:
|
|
|
|
1. Elimination of the layers feature, and with it the ability to place
|
|
images behind text. In this proposal, a Cell on the screen will
|
|
show either a (part of a) visible image, or a (part of a) text
|
|
glyph, but never both.
|
|
|
|
2. Elimination of the "url" option, and with it the ability for an
|
|
application to specify a filename or other method for the terminal
|
|
to find the file data on the local machine. Image data must always
|
|
be passed inline with the sequences.
|
|
|
|
3. Elimination of response codes, and with it:
|
|
|
|
- The ability for multiplexers to blindly pass on the sequences to
|
|
their host terminal (because unique IDs are not generated by the
|
|
terminal).
|
|
|
|
- The ability for applications to reliably detect success or
|
|
failure of image display operations.
|
|
|
|
4. Elimination of pixel-oriented image placement operations, and with
|
|
it the ability of applications to pass on image calculations to the
|
|
terminal. An application which requires pixel-perfect rendering
|
|
must generate the pixels it needs, aligned such to be displayed at
|
|
the top-left corner of the text Cell rectangle.
|
|
|
|
|
|
|
|
Summary
|
|
-------
|
|
|
|
This revised document proposes two independent new features:
|
|
|
|
1. A method to transfer image data for immediate display within the
|
|
screen Cell grid ("Direct Images").
|
|
|
|
2. A method to transfer image data to a terminal-managed cache, and
|
|
later display that data within the screen Cell grid ("Cached
|
|
Images").
|
|
|
|
The only difference between the first and second feature is the
|
|
presence of an ID key. Direct images do not use an ID key, while
|
|
cached images use a store operation with ID key followed by one or
|
|
more display operations with ID key.
|
|
|
|
Images are applied to text Cells, and once set handled the same way
|
|
text Cells are handled: erasing a line erases the image Cells on that
|
|
line, inserting a character will shift image Cells on that row over,
|
|
scrolling will shift the image up, and so on. Therefore, terminals
|
|
will need to be prepared for the scenario that every Cell on the
|
|
display is a separate image, with a separate display scaling option
|
|
that will need to be re-applied automatically if font metrics change.
|
|
|
|
|
|
|
|
All Features - Detection
|
|
------------------------
|
|
|
|
Applications can detect support for these features using Primary
|
|
Device Attributes (DA) and DECID (ESC Z, or 0x9A).
|
|
|
|
Terminals that support this standard will repond with additional
|
|
parameter(s): "224" for direct images and "225" for cached images. A
|
|
recap of the parameters xterm supports is listed below, with these new
|
|
feature responses included:
|
|
|
|
| VT220 (and higher) Response | Description |
|
|
|-----------------------------|--------------------------------------------|
|
|
| 1 | 132-columns |
|
|
| 2 | Printer |
|
|
| 3 | ReGIS graphics |
|
|
| 4 | Sixel graphics |
|
|
| 6 | Selective erase |
|
|
| 8 | User-defined keys |
|
|
| 9 | National Replacement Character sets |
|
|
| 1 5 | Technical characters |
|
|
| 1 6 | Locator port |
|
|
| 1 7 | Terminal state interrogation |
|
|
| 1 8 | User windows |
|
|
| 2 1 | Horizontal scrolling |
|
|
| 2 2 | ANSI color, e.g., VT525 |
|
|
| 2 8 | Rectangular editing |
|
|
| 2 9 | ANSI text locator (i.e., DEC Locator mode) |
|
|
| 2 2 4 | Direct Images Version 1 |
|
|
| 2 2 5 | Cached Images Version 1 |
|
|
|
|
|
|
|
|
Direct Images - Summary
|
|
-----------------------
|
|
|
|
Non-text data (images) can be sent to the terminal for immediate
|
|
display in a rectangular region of text Cells. Image data is
|
|
transmitted to the terminal using a wire format described later in
|
|
this document.
|
|
|
|
Setting a Cell to image is a destructive operation: the Cell's
|
|
original text is lost. Similarly, setting a Cell (or multiple Cells
|
|
for fullwidth glyphs or grapheme clusters) to text is a destructive
|
|
operation: the image in the Cell(s) is lost.
|
|
|
|
Setting any part of a multi-Cell Tile to image also "breaks up" the
|
|
Tile into a range of single Cells. In other words, image data can
|
|
only be carried by a Cell, not a Tile.
|
|
|
|
|
|
|
|
Direct Images - New Sequences
|
|
-----------------------------
|
|
|
|
A terminal with direct images feature must support the following new
|
|
sequences:
|
|
|
|
| Sequence | Description |
|
|
|--------------------------------------|-------------------------|
|
|
| OSC 1 3 3 8 ; F i l e = {args} : {data} BEL | Display image at (x, y) |
|
|
| OSC 1 3 3 8 ; F i l e = {args} : {data} ST | Display image at (x, y) |
|
|
|
|
|
|
|
|
For the OSC 1 3 3 8 sequence:
|
|
|
|
* The {args} is a set of key-value pairs (each pair separated by
|
|
semicolon (';')), followed by a colon (':'), followed by a base-64
|
|
encoded string ({data}).
|
|
|
|
* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
|
|
'a' - 'z').
|
|
|
|
* A value is any printable ASCII string not containing whitespace,
|
|
colon, or semicolon ('!' - '9', '<' - '~').
|
|
|
|
* Any alpha-numeric key may be specified. A key that is not supported
|
|
by the terminal is ignored without error.
|
|
|
|
* The image is processed as shown below:
|
|
|
|
- The pixels are drawn starting at the upper-left corner of the text
|
|
cursor position.
|
|
|
|
- All pixels in the target Cell rectangle that are not covered by
|
|
the image itself are set the current background color (like
|
|
sixel raster attributes).
|
|
|
|
- If scroll is specified as 1 (enabled), then:
|
|
|
|
a. The screen is scrolled up if the image overflows into the
|
|
bottom text row.
|
|
|
|
b. The cursor's final position is on the same column as the
|
|
starting cursor position, and on the row immediately below the
|
|
image.
|
|
|
|
- If scroll is omitted or specified as 0 (disabled), then:
|
|
|
|
a. The screen is never scrolled.
|
|
|
|
b. Pixels that would be drawn below the visible region on screen
|
|
are discarded.
|
|
|
|
c. The cursor's final position is at the same column and row as
|
|
the starting cursor position, i.e. the cursor does not move at
|
|
all.
|
|
|
|
- Pixels that would be drawn to the right of the visible region on
|
|
screen are discarded.
|
|
|
|
- If scale is "none", then pixels that would be drawn outside the
|
|
target Cell rectangle are discarded.
|
|
|
|
|
|
|
|
The keys for the key-value pairs that must be supported by the
|
|
terminal are listed below:
|
|
|
|
| Key | Default Value | Description |
|
|
|--------------|---------------|---------------------------------------|
|
|
| type | "image/rgb" | mime-type describing data field |
|
|
| width | 1 | Number of Cell columns to display in |
|
|
| height | 1 | Number of Cells rows to display in |
|
|
| scale | "none" | Scale/zoom option, see below |
|
|
| sourceX | 0 | Media source X position to display |
|
|
| sourceY | 0 | Media source Y position to display |
|
|
| sourceWidth | "auto" | Media width in pixels to display |
|
|
| sourceHeight | "auto" | Media height in pixels to display |
|
|
| scroll | 0 | If 0, scroll the display if needed |
|
|
|
|
A terminal may support additional keys. If a key is specified but not
|
|
supported by the terminal, then it is ignored without error.
|
|
|
|
|
|
|
|
The "type" value is a mime-type string describing the format of the
|
|
base64-encoded binary data. The terminal must support at minimum these
|
|
mime-types:
|
|
|
|
| Type String | Description |
|
|
|---------------|--------------------------------------------------------------|
|
|
| "image/rgb" | Big-endian-encoded 24-bit red, green, blue values |
|
|
| "image/rgba" | Big-endian-encoded 32-bit red, green, blue, alpha values |
|
|
| "image/png" | PNG file data as described by (reference to PNG format) |
|
|
|
|
A terminal may support additional types. An application can detect
|
|
terminal support for a format by:
|
|
|
|
1. Attempt to draw image, with "scroll" set to 1.
|
|
|
|
2. Check cursor position DSR 6.
|
|
|
|
3. If cursor has moved, then the terminal supports this image type.
|
|
|
|
|
|
|
|
The "width" and "height" values are positive integers describing the
|
|
number of Cells the image will be placed in.
|
|
|
|
|
|
|
|
The "scale" value can take the following values:
|
|
|
|
| Value | Meaning |
|
|
|------------|---------------------------------------------------------------|
|
|
| "none" | No scaling along either axis. |
|
|
| "scale" | Stretch image, preserving aspect ratio, to maximum size in the target area without cropping |
|
|
| "stretch" | Stretch along both axes, distorting aspect ratio, to fill the target area |
|
|
| "crop" | Stretch along both axes, preserving aspect ration, to completely fill the target area, cropping pixels that will not fit |
|
|
|
|
|
|
|
|
"sourceX", "sourceY", "sourceWidth", and "sourceHeight" define the
|
|
rectangle of pixels from the media that will be displayed on the
|
|
screen. The ranges for these values is shown below:
|
|
|
|
| Key | Minimum Value | Maximum Value | Default Value |
|
|
|--------------|---------------|-------------------------------|---------------|
|
|
| sourceX | 0 | Media's full width - 1 | 0 |
|
|
| sourceY | 0 | Media's full height - 1 | 0 |
|
|
| sourceWidth | 1 | Media's full width - sourceX | "auto" |
|
|
| sourceHeight | 1 | Media's full height - sourceY | "auto" |
|
|
|
|
If any of these values are specified and outside the range, no image
|
|
is displayed, and the cursor does not move. "sourceWidth" and
|
|
"sourceHeight" can be "auto", which means use the maximum available
|
|
width/height (given sourceX/sourceY) from the media's inherent
|
|
dimensions.
|
|
|
|
|
|
|
|
Cached Images - Summary
|
|
-----------------------
|
|
|
|
Non-text data (image) can be sent to the terminal for later display in
|
|
a rectangular region of text Cells. Image data is transmitted to the
|
|
terminal using the CSTORE command described below, and displayed on
|
|
screen using the CDISPLAY command. A single CSTORE command can
|
|
support many CDISPLAY commands.
|
|
|
|
Upon display, setting a Cell to image is a destructive operation: the
|
|
Cell's original text is lost. Similarly, setting a Cell (or multiple
|
|
Cells for fullwidth glyphs or grapheme clusters) to text is a
|
|
destructive operation: the image in the Cell(s) is lost.
|
|
|
|
Setting any part of a multi-Cell Tile to image also "breaks up" the
|
|
Tile into a range of single Cells. In other words, image data can
|
|
only be carried by a Cell, not a Tile.
|
|
|
|
|
|
|
|
Cached Images - Cache/Memory Management
|
|
---------------------------------------
|
|
|
|
The terminal manages a cache of multimedia data on behalf of the
|
|
application. The application requests media be stored in the cache
|
|
and provides an ID. This ID is later used to request display on the
|
|
screen.
|
|
|
|
The amount of memory and retention/eviction strategy for the cache is
|
|
wholly managed by the terminal, with the following restrictions:
|
|
|
|
* The terminal may not remove items from the cache that have any
|
|
portion being actively displayed on the primary or alternate
|
|
screens.
|
|
|
|
The scrollback buffer is permitted, and recommended, to contain only a
|
|
few (or zero) multimedia images. Terminals should consider retaining
|
|
only the last 2-5 screens' worth of pixel data in the scrollback
|
|
buffer.
|
|
|
|
Applications have no control over when images are removed from the
|
|
cache, and no provision is made to generate/ensure unique IDs.
|
|
|
|
A terminal multiplexer that passes all CSTORE/CDISPLAY commands to the
|
|
host terminal will need to parse the CSTORE and CDISPLAY sequences for
|
|
the "id" field and rewrite it to be unique for all of its inner
|
|
terminals.
|
|
|
|
|
|
|
|
Cached Images - New Sequences
|
|
-----------------------------
|
|
|
|
A terminal with cached images feature must support the following new
|
|
sequences:
|
|
|
|
| Sequence | Command | Description |
|
|
|--------------------------------------|-----------|--------------------------|
|
|
| OSC 1 3 4 0 ; F i l e = {args} : {data} BEL | CSTORE | Store image in cache |
|
|
| OSC 1 3 4 0 ; F i l e = {args} : {data} ST | CSTORE | Store image in cache |
|
|
| OSC 1 3 4 1 ; Pi ; {args} BEL | CDISPLAY | Display image at (x, y) |
|
|
| OSC 1 3 4 1 ; Pi ; {args} ST | CDISPLAY | Display image at (x, y) |
|
|
|
|
|
|
|
|
Cached Images - CSTORE
|
|
----------------------
|
|
|
|
For the CSTORE command:
|
|
|
|
* The {args} is a set of key-value pairs (each pair separated by
|
|
semicolon (';')), followed by a colon (':'), followed by a base-64
|
|
encoded string ({data}).
|
|
|
|
* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
|
|
'a' - 'z').
|
|
|
|
* A value is any printable ASCII string not containing whitespace,
|
|
colon, or semicolon ('!' - '9', '<' - '~').
|
|
|
|
|
|
|
|
The keys for the key-value pairs that must be supported by the
|
|
terminal are listed below:
|
|
|
|
| Key | Default Value | Description |
|
|
|--------------|---------------|----------------------------------------------|
|
|
| id | 0 | ID to refer to the image |
|
|
| type | "image/rgb" | mime-type describing data field |
|
|
|
|
|
|
|
|
The "id" value is a non-negative integer between 0 and 999999.
|
|
|
|
|
|
|
|
The "type" value is a mime-type string describing the format of the
|
|
base64-encoded binary data. The terminal must support at mimunum these
|
|
mime-types:
|
|
|
|
| Type String | Description |
|
|
|---------------|--------------------------------------------------------------|
|
|
| "image/rgb" | Big-endian-encoded 24-bit red, green, blue values |
|
|
| "image/rgba" | Big-endian-encoded 32-bit red, green, blue, alpha values |
|
|
| "image/png" | PNG file data as described by (reference to PNG format) |
|
|
|
|
A terminal may support additional types. An application can detect
|
|
terminal support for a format by:
|
|
|
|
1. Store image in cache.
|
|
|
|
2. Attempt to draw image, with "scroll" set to 1.
|
|
|
|
3. Check cursor position DSR 6.
|
|
|
|
4. If cursor has moved, then the terminal supports this image type.
|
|
|
|
|
|
|
|
Cached Images - CDISPLAY
|
|
------------------------
|
|
|
|
For the CDISPLAY command:
|
|
|
|
* Pi - a non-negative integer ID that was used in a previous CSTORE
|
|
command.
|
|
|
|
* The {args} is a set of key-value pairs (each pair separated by
|
|
semicolon (';')), followed by a colon (':'), followed by a base-64
|
|
encoded string.
|
|
|
|
* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
|
|
'a' - 'z').
|
|
|
|
* A value is any printable ASCII string not containing whitespace,
|
|
colon, or semicolon ('!' - '9', '<' - '~').
|
|
|
|
* Any alpha-numeric key may be specified. A key that is not supported
|
|
by the terminal is ignored without error.
|
|
|
|
* The image pixels are processed as shown below.
|
|
|
|
- The pixel are drawn starting at the upper-left corner of the text
|
|
cursor position.
|
|
|
|
- If scroll is specified as 1 (enabled), then:
|
|
|
|
a. The screen is scrolled up if the image overflows into the
|
|
bottom text row.
|
|
|
|
b. The cursor's final position is on the same column as the
|
|
starting cursor position, and on the row immediately below the
|
|
image.
|
|
|
|
- If scroll is omitted or specified as 0 (disabled), then:
|
|
|
|
a. The screen is never scrolled.
|
|
|
|
b. Pixels that would be drawn below the visible region on screen
|
|
are discarded.
|
|
|
|
c. The cursor's final position is at the same column and row as
|
|
the starting cursor position, i.e. the cursor does not move at
|
|
all.
|
|
|
|
- Pixels that would be drawn to the right of the visible region on
|
|
screen are discarded.
|
|
|
|
|
|
|
|
The keys for the key-value pairs that must be supported by the
|
|
terminal are listed below:
|
|
|
|
| Key | Default Value | Description |
|
|
|--------------|---------------|---------------------------------------|
|
|
| id | 0 | ID to refer to the image |
|
|
| width | 1 | Number of Cell columns to display in |
|
|
| height | 1 | Number of Cells rows to display in |
|
|
| scale | "none" | Scale/zoom option, see below |
|
|
| sourceX | 0 | Media source X position to display |
|
|
| sourceY | 0 | Media source Y position to display |
|
|
| sourceWidth | "auto" | Media width in pixels to display |
|
|
| sourceHeight | "auto" | Media height in pixels to display |
|
|
| scroll | 0 | If 1, scroll the display if needed |
|
|
|
|
A terminal may support additional keys. If a key is specified but not
|
|
supported by the terminal, then it is ignored without error.
|
|
|
|
|
|
|
|
The "width" and "height" values are positive integers describing the
|
|
number of Cells the image will be placed in.
|
|
|
|
|
|
|
|
The "scale" value can take the following values:
|
|
|
|
| Value | Meaning |
|
|
|------------|---------------------------------------------------------------|
|
|
| "none" | No scaling along either axis. |
|
|
| "scale" | Stretch image, preserving aspect ratio, to maximum size in the target area without cropping |
|
|
| "stretch" | Stretch along both axes, distorting aspect ratio, to fill the target area |
|
|
| "crop" | Stretch along both axes, preserving aspect ration, to completely fill the target area, cropping pixels that will not fit |
|
|
|
|
|
|
|
|
"sourceX", "sourceY", "sourceWidth", and "sourceHeight" define the
|
|
rectangle of pixels from the media that will be displayed on the
|
|
screen. The ranges for these values is shown below:
|
|
|
|
| Key | Minimum Value | Maximum Value | Default Value |
|
|
|--------------|---------------|-------------------------------|---------------|
|
|
| sourceX | 0 | Media's full width - 1 | 0 |
|
|
| sourceY | 0 | Media's full height - 1 | 0 |
|
|
| sourceWidth | 1 | Media's full width - sourceX | "auto" |
|
|
| sourceHeight | 1 | Media's full height - sourceY | "auto" |
|
|
|
|
If any of these values are specified and outside the range, no image
|
|
is displayed, and the cursor does not move. "sourceWidth" and
|
|
"sourceHeight" can be "auto", which means use the maximum available
|
|
width/height (given sourceX/sourceY) from the media's inherent
|
|
dimensions.
|
|
|
|
|
|
|
|
Miscellaneous Items
|
|
-------------------
|
|
|
|
"image/rgb" and "image/rgba" also need width/height fields. Propose
|
|
to specify them as 16-bit unsigned ints, followed by 24-bit or 32-bit
|
|
data. If data is short, then the rest of the image is assumed to be
|
|
current background color (like sixel raster attributes).
|