Scripting with ANSI Color Codes

Oh the console! Love it or loath it, if you engage in computer actions you probably end up spending a lot of time looking at a console.

Of course not all consoles are created equal. Some consoles are really just the equivalent of

while line=nextLine() ; do print line ; end

Typically this is reserved to crappy IDE’s (yes, I know that’s redundant, I’m making a hah-hah).

Some deluded souls think cygwin is a real console, only to discover half-way into trying to actually get something done that it ends up going nutso and spewing text in the most unlikely of locations. Infrequently this can lead to a series of awkward questions in the ER.

Gnome’s terminal is pretty adequate, and of course you can’t go wrong with xterm or a host of other well written, robust terminals that oddly enuff seem to show up only under flavors of UNIX (yes, netBSD… err.. Mac counts too).

Why is writing a solid console implementation so difficult?

Because a console is not about printing lines of text. A terminal is more like a pixel-based display where ever pixel is a character + presentation information.

It is a highly sophisticated user interface. Unlike the farktarded point-and-click UI’s popular today for no good reason which use a maximum of say 8 buttons, a console typically has 108+ buttons.

Hah-hah, indeed. More importantly a console is really an interpreter.

No, I don’t mean the shell, I mean the console is an interpreter. Just like with shells we have bash, tcsh and crapsh, terminals come in flavors like VT100, VT110, VT220, 3270, 5270 and plethora of others.

Since a terminals has to accept all sorts of input, the language for controlling the terminal is a done via “control codes.” Typically a control code is just a string which is difficult to type.

For example: echo <ESC>[33mCheese<ESC>[0m

What is that <ESC>? Typically it is character 27. How do you type it? On a lot of consoles, you type it like this:

  1. hold down the control key
  2. press the ‘v’ key
  3. let go of the control and ‘v’ key
  4. press the escape key

Pretty neat, huh? That works for ANSI/VT100-based terminals. Not for all terminals. Luckily for us, that’s just about all we really care about.

Of course a real console application (eg: vim) cares about all terminals and you start to move away from writing control codes directly and towards using terminal apis like curses/ncurses PDQ.

For hacking purposes, VT100 terminal codes will do just fine.

I know.. what a load of historic irrelevant blather and so what and quit wasting my time! What is it good for?!

When was the last time you looked at a log file? Let me put it a different way, when did you stop looking at log files? Chances are you are watching a log file right now! I know I am!

What percentage of log output do you care about for a given task? 10%? 2%? Isn’t it a drag that it just blends in with the rest?

Of course you could grep it out, but then you lose the context… Or you might forget a term and have to reproduce the problem again (again [again]).

IMHO, a kick-aspirin use for VT100 color codes is for log highlighting, and it is some simple, it can be easily scripting into a general use script.

Here is the kind of usage I would like:

usage: cat foo | line_lite.sh ( pattern highlighting )+

pattern         is a ala grep
highlighting    is a colon separated list of colors

the colors are 

    reset        bright      dim         underline    blink
    reverse      hidden      black       red          green
    yellow       blue        magenta     cyan         white
    black_bg     red_bg      green_bg    yellow_bg    blue_bg
    magenta_bg   cyan_bg     white_bg

That way I can use it like: tail -f some.log | line_lite.sh Exception red “some other text” green:underline

In order to make this happen, I need to convert that red, green:underline crap to the appropriate VT100 code.

Natually, I do this with sed:

_line_lite_color_to_code() {
    echo ${*} | sed "s,reset,0,g;s,bright,1,g;s,dim,2,g;s,underline,4,g;s,blink,5,g;s,reverse,7,g;s,hidden,8,g;s,black,30,g;s,red,31,g;s,green,32,g;s,yellow,33,g;s,blue,34,g;s,magenta,35,g;s,cyan,36,g;s,white,37,g;s,black_bg,40,g;s,red_bg,41,g;s,green_bg,42,g;s,yellow_bg,43,g;s,blue_bg,44,g;s,magenta_bg,45,g;s,cyan_bg,46,g;s,white_bg,47,g;s,:,;,g"
}

_line_lite_code() {
    _line_lite_color_to_code ${*} | sed 's,.*,ESC[&m,'
}

This will convert “green:underline” to [32;4m

Now we just need some way to do our matching and add the color code. Once again, it’s sed time.

For each pair of pattern/color, we create a sed expression like: s!.*.*!&!g;

In this case we’d have s!.*some other text.*![32;4m&[0m!g;

All that we have to do is concatenate our sed expressions for each pattern/color pair and then call sed.

Viola! Arbitrarily colored logs!

Here is sample output from something that’s not a log:

% jar_minder_javap sun.security.provider.PolicyFile | line_lite.sh 'static'  red:bright
Compiled from "PolicyFile.java"
public class sun.security.provider.PolicyFile extends java.security.Policy{
    public sun.security.provider.PolicyFile();
    public sun.security.provider.PolicyFile(boolean);
    public void refresh();
    public boolean implies(java.security.ProtectionDomain, java.security.Permission);
    public java.security.PermissionCollection getPermissions(java.security.ProtectionDomain);
    public java.security.PermissionCollection getPermissions(java.security.CodeSource);
    protected java.security.cert.Certificate[] getSignerCertificates(java.security.CodeSource);
    static boolean access$002(sun.security.provider.PolicyFile, boolean);
    static boolean access$102(sun.security.provider.PolicyFile, boolean);
    static boolean access$202(sun.security.provider.PolicyFile, boolean);
    static boolean access$302(sun.security.provider.PolicyFile, boolean);
    static boolean access$200(sun.security.provider.PolicyFile);
    static sun.security.util.Debug access$400();
    static boolean access$500(sun.security.provider.PolicyFile, java.net.URL, sun.security.provider.PolicyInfo);
    static java.security.CodeSource access$600(sun.security.provider.PolicyFile, java.security.CodeSource, boolean);
    static java.lang.String access$700(sun.security.provider.PolicyFile, java.security.ProtectionDomain);
    static java.security.IdentityScope access$800();
    static {};
}

Sadly, my olde ascii2html script is not quite it… but I think you get the idea.

Go forth and dig them control codes, my friend! They exist only to serve your dread will.

Here is a link to a copy of the script just in case: http://brianin3d.googlepages.com/line_lite.sh


Dude has had this great link for a jillion years! http://www.fh-jena.de/~gmueller/Kurs_halle/esc_vt100.html, what a hero!

P.S.: Yes, I advocate EMACS-hateration. set -o vi or fight!

Advertisements

11 Comments »

  1. Vic said

    With a careful eye to distinguish ” from ” one can get the script to work, though alas in my gnome-terminal my carefully highlighted lines look like,

    ESC[31;1m2007-12-18 16:26:41,772 DEBUG [org.jboss.mq.il.uil2.SocketManager] Failed to send error replyESC[0m

    – oom

  2. brianin3d said

    Be sure that in the function:

    line_lite_code() {
    _line_lite_color_to_code ${*} | sed ‘s,.*,ESC[&m,’
    }

    That you have a real escape character and not just the letters ‘E’, ‘S’ and ‘C’.

    Here is a link to a copy of the script just in case: http://brianin3d.googlepages.com/line_lite.sh

  3. Craig said

    Great post. 100% what I was looking for. Thanks for explaining the escape characters.

  4. brianin3d said

    Sometimes you may get sick of typing: v[31m , perhaps you will find this useful:

    wget http://brianin3d.googlepages.com/vt100.colors.sh
    source vt100.colors.sh
    echo ${vt_green}${bg_black}terminal coloring is neat${vt_none}

    Feel free to use it however you want, but I would appreciate it if you would use it to kick some ass… 😉

  5. Anonymous Coward said

    I’m sorry but saying that a terminal is pixel-based is completely wrong. The whole point of terminals is to be able to send characters down the line, not pixels. If it was pixel-based perfs would be pathetic.

    character+code, yup.

    pixel+codes, definitely not. This is completely bogus thinking.

    Entirely missing the point.

  6. brianin3d said

    Hmm… I guess… you need some additional explanation.

    The mention of pixels is an analogy… I’m not saying the terminal is a actually a collection of pixels.

    I’m saying that a console is comprised of molecular units which are control atom and character atoms…

    Just so you don’t get spaced out again… I am not saying the terminal is the periodic table… nor when I say “get spaced out” I don’t think you are an astronaut.

    Talk about entirely missing the point. Haha.

  7. jangxyz said

    thanks. i wanted to give color to the output in character-base, and your post helped!

    a simple code that explains the idea clearly.

    i also thought of merging the two, but too much bash’ing and sed’ing won’t be good for my soul 😉

  8. SD Snatcher said

    Great script! It’s just what I was needing! Tks!

    I just have a small bug-report:

    It works fine when I type:
    $ echo test | line_lite.sh ‘test’ red
    Result: word “test” is printed in red

    But OTOH, it gives me the wrong output when I try any of the “_bg” color:
    $ echo test | line_lite.sh ‘test’ red_bg
    Result: Prints “bgmtest” without setting the color

  9. brianin3d said

    Yeh… that’s definitely my bad…. I think I used _ as the delim to sed… pretty silly…

    These days, I just source this: http://code.google.com/p/brianin3d-misc/source/browse/trunk/scripts/vt100.colors.sh

    And do the sed inline… I should fix the script though…

  10. cdf said

    If you want the _bg colours to work, you’ll need to reorder the sed line in _line_lite_color_to_code so that the _bg colors are replaced before the non-_bg ones.

    EG. Currently the ‘red’ in ‘red_bg’ will be replaced first, leaving an invalid color code.

    Very helpful script though, thankyou!

  11. brianin3d said

    Thanks, man… I’ve been meaning to fix it for a long time… just lazy…

    Here is a bonus trick to use vt100 codez to set the title of a terminal:

    set_title () {
            echo -e "CRAP033]0;" ${*} "CRAP007"
    }
    

    Wow… forgot how much trying to type stuff like that sux in wordpress… that “CRAP” should be replaced with a backslash

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: