Posts Tagged scripting

making m2 salad

Irrelevant Silliness

So… I’m a vim hold out. I still care deeply about emacs and it’s spawn being defeated once and for all.

Which means I don’t really like bloated IDE’s like Eclipse…. which makes me a pariah… which means people whisper and plot against me behind my back…

And… they’re always out to get me… and…

Ahem… sorry, about that… Like I was saying….

Which means I prefer to write a lot of tools and utilities to provide the sort of out-of-the-box functionality other people are perfectly happy to point and click thru with their brains in the neutral gear.

It’s part of a sickness I have… and I am going to confess it here on the internet: I love programming computers.

I know, I know… appalling! The gall! But there it is! My dirty little secret!

So what have I brewed up this time?

You know that functionality which lets you see right click and see methods, etc in Eclipse?

I thought it would be nice to have something like it for the CLI where I can use grep the way the Lawd intended. Here is the soup-to-nuts for nuts.

Finally he gets to the point

% m2_salad.sh 
m2_cp.txt is older than pom.xml... rebuilding m2_cp.txt
m2_soup.txt is older than m2_cp.txt... rebuilding m2_soup.txt
m2_salad.txt is older than m2_soup.txt... rebuilding m2_salad.txt

Zounds! So I wrote 2 new scripts: m2_soup.sh and m2_salad.sh.

Like m2_cp.sh, each produces a file named X.txt with some junk in it.

Here is the breakdown:

m2_cp.txt      colon separated list of the jars maven uses
m2_soup.txt    list of all the classes defined in the m2_cp.txt jars
m2_salad.txt   javap of all the classes listed in m2_soup.txt

These along with the standard UNIX CLI utils provides the same sort of handy lookup funtionality, where it belongs: in the CLI, not in the editor.

And?

Well… OK, maybe it could live in the editor too if it was slim-and-fast and minimalist…

I’ve really enjoyed the vim plugin supertab for tab completion.

Part of the challenge of integrating it with the output from the scripts is to make the completion be contextual… Which means a plugin would have to have some notion of the program semantics.

For example, it would need to know that a piece of text was a variable and what it’s type was so it could lookup the appropriate methods.

Not easy, but I’m a big believer in the 90/10 rule.

Alternatively… what other sort of useful development application could be written leveraging this info?

Good times… good times…

Advertisements

Leave a Comment

Print the name of any class in a jar that has a main method:

Print the name of any class in a jar that has a main method:

jar="output/opennlp-tools-1.4.3.jar"
javap -classpath ${jar} $(     
    jar tf ${jar}      \
    | grep ass$        \
    | cut -f1 -d.      \
    | tr '[$/]' .      
) \ 
| tr '\n' '@'          \
| sed 's,}@,}=,g'      \
| tr '=' '\n'          \
| grep -w main         \
| tr '@' '\n'          \
| grep '^public class' \
| awk '{print $3}'

Leave a Comment

pull analytics “site search terms” with google data api

I’d heard about the google data api, but up until recently hadn’t found much practical use for it.

Suddenly I needed to pull back the top search terms for a site monitored by google analytics. The default view only shows 500 at a time. Granted this accounted for 30% of the “unique searches’, but I wanted to see a bit more.

Though it was sort of hard to find, I used the example script dataFeed.sh to retrieve the top 10k search terms.

The script wants you to set the variables for username/password and “PROFILE_ID” The value for this last is the id=xxxx in the url when you cruise around the analytics site.

Rather than read thru the quirky documentation as to what dimensions and metrics were available, I used the Data Feed Query Explorer to find the values instead:

feedUri="https://www.google.com/analytics/feeds/data\
?start-date=2009-08-01\
&end-date=2009-12-04\
&dimensions=ga:searchKeyword\
&metrics=ga:searchDepth,ga:searchDuration,ga:searchExits,ga:searchRefinements,ga:searchUniques,ga:searchVisits\
&sort=-ga:searchUniques\
&max-results=40000\
&ids=ga:$PROFILE_ID\
&prettyprint=true"

curl $feedUri -s --header "Authorization: GoogleLogin $googleAuth"

Despite the max-results of 40k, it looks like there is a max of 10k, which is fine.

All in all, pretty ez. Digging it all up.. sort of sucked.

Hence the write up… hope it helps some buddy!

Leave a Comment

4 useful shell scripts for maven

yeah maven

Maven (2 of course) is the jam when it comes to Java build systems.

Ant is some old timey donkey-mess madness which doesn’t even address the issue of dynamic build dependency resolution.

There are other build systems like: ivy, maven1, and made-up-mess, but the real goodness is the m2 universe.

In addition, to it’s main duties, maven produces a really neat resource in the form of your local repository (mine lives in ${HOME}/.m2/repository).

The dependencies your project pulls in also pull in dependencies and some of them are cool things you didn’t even know you couldn’t live without!

Here are a few scripts I use to leverage a little bit of that powerful force for funtimes!

m2_cp.sh : maven classpath

Maven compiles all your code and runs all your test cases. To do that it has to use an olde timey classpath just like duh, but …. how can you get at it when all you see is your pom.xml?

You can use the olde school trick: mvn -X test -Dmaven.test.skip=true ; and scrap it out of there

Or… you can use my little m2_cp.sh and it will parse it out for you into a file called “m2_cp.txt” that you can use to play more reindeer games.

Like what?

Like packaging up some tarball+script to run your stuff, or some uberJar (though the assembly plugin is a better solution for that), or maybe some…

m2_javap.sh : all your class are… ok…

Almost every Java developer uses an IDE. They pretty much all use eclipse.

I’m in that oddball 0.01% that just uses eclipse as a remote debugger… so… I use a lot of command line tools instead, for example: javap

The javap command will print the method signatures given a list of class names. But… you have to feed it a classpath.

In tricky eclipse, this happens like magic! Neat! But… as a magician, I want to be the magical man… so…

When I’m working on say, a portlet project (from maven-archetype-portlet), and I’m wanting to see some good times, I type: m2_javap.sh javax.portlet.GenericPortlet

Even that too much bother? How ’bout: m2_javap.sh GenericPortlet

m2_latest.sh : to the greatest

Ever use log4j in a project? Ha! Yeah… so now you have a new project and you want to pull it in again… what as the most recent version: m2_latest.sh log4j ; tells you the jar

m2_dep.sh : jar file to dependency

OK, so you need log4j, and the latest is ${HOME}/.m2/repository/log4j/log4j/1.2.14/log4j-1.2.14.jar ; time to start typing that annoying stanza… but wait:

m2_latest.sh log4j | m2_dep.sh

% m2_latest.sh log4j | m2_dep.sh

<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.14</version>
</dependency>

Script it up

The local repo is really neato! Perfect scripting fodder! Good times…

Leave a Comment

scripting essentials: fullpath

One of the things you end up doing in cli world a lot is jacking around with files.

Usually you are doing some junk with files in some (collection of) terminal(s) with a given working directory and referring to files with a local path eg: src/main/webapp/WEB-INF/web.xml

Then… for some reason, you want to manipulate them from another context, maybe a browser or some shells doing stuff in another working directory. In some cases you could change working directories, but is that really what you want?

Kind of breaks the flow, doesn’t it? “Now let me hop over here…”

Why are you hopping? You want to grep src/main/webapp/WEB-INF/web.xml? Just use the fullpath name, eg: /home/jameson/src/devilish/src/main/webapp/WEB-INF/web.xml !

But how? EZ: echo ${PWD}/src/main/webapp/WEB-INF/web.xml

By jamming around the fullname instead of hopping around the filesystem like a bunny rabbit, you will be better able to maintain the separation between the different contexts of your activities.

Ya dig it? If not, try this approach out for a while and see if you can feel the difference.

To make this easier, I prefer to use a bash function like this:

fullpath() { 
    local f
    for f in ${*}; do
        ( 
		 	( 
			 	cd ${f} && echo ${PWD} 
			) || ( 
			 	cd $(dirname ${f} ) && echo ${PWD}/$( basename ${f} ) 
			) 
		) 2> /dev/null
    done
}

This takes care of some annoying corner cases that I can’t remember at the moment… but feel free to shorten it

This tip says: move the world around your mind, not your mind around the world.

Leave a Comment

using java.util.jar.JarFile

Of course Java can read zip files / jars. It’s pretty straightforward:

import java.util.jar.JarFile;
import java.util.Enumeration;
//...
    public void print( String arg ) throws Exception {
        this.print( new JarFile( arg ) );
    }

    public void print( JarFile jarFile ) {
        this.print( jarFile.entries()  );
    }

    public void print( Enumeration entries ) {
        while ( entries.hasMoreElements() ) {
            System.out.println( entries.nextElement() );
        }
    }

program, compile and execute thyself!

I wrote a little test program called Jarout.java, which is a “self-compiling” program:

% ./Jarout.java ~/.m2/repository/org/springframework/spring/2.5.5/spring-2.5.5.jar

This works (in reasonable environments) because the first 4 lines are a shell script protected by a block comment:

/*2222222 2>/dev/null
javac Jarout.java && java Jarout ${*}
exit ${?}
*/

import java.util.jar.JarFile;
import java.util.Enumeration;
//...

The first line keeps the shell quiet and the exit keeps the rest of the program from being interpreted. It’s a silly trick, but useful sometimes. I also use it for C and C++. The gcc is so fast that it gives traditional interpreted languages a run for their money!

Even though it is a silly trick, it allows you to use any compiled language like a scripting language…

😀

Leave a Comment

Scripting with ANSI Color Codes

Oh the console! Love it or loath it, if you engage in computer actions you probably end up spending a lot of time looking at a console.

Of course not all consoles are created equal. Some consoles are really just the equivalent of

while line=nextLine() ; do print line ; end

Typically this is reserved to crappy IDE’s (yes, I know that’s redundant, I’m making a hah-hah).

Some deluded souls think cygwin is a real console, only to discover half-way into trying to actually get something done that it ends up going nutso and spewing text in the most unlikely of locations. Infrequently this can lead to a series of awkward questions in the ER.

Gnome’s terminal is pretty adequate, and of course you can’t go wrong with xterm or a host of other well written, robust terminals that oddly enuff seem to show up only under flavors of UNIX (yes, netBSD… err.. Mac counts too).

Why is writing a solid console implementation so difficult?

Because a console is not about printing lines of text. A terminal is more like a pixel-based display where ever pixel is a character + presentation information.

It is a highly sophisticated user interface. Unlike the farktarded point-and-click UI’s popular today for no good reason which use a maximum of say 8 buttons, a console typically has 108+ buttons.

Hah-hah, indeed. More importantly a console is really an interpreter.

No, I don’t mean the shell, I mean the console is an interpreter. Just like with shells we have bash, tcsh and crapsh, terminals come in flavors like VT100, VT110, VT220, 3270, 5270 and plethora of others.

Since a terminals has to accept all sorts of input, the language for controlling the terminal is a done via “control codes.” Typically a control code is just a string which is difficult to type.

For example: echo <ESC>[33mCheese<ESC>[0m

What is that <ESC>? Typically it is character 27. How do you type it? On a lot of consoles, you type it like this:

  1. hold down the control key
  2. press the ‘v’ key
  3. let go of the control and ‘v’ key
  4. press the escape key

Pretty neat, huh? That works for ANSI/VT100-based terminals. Not for all terminals. Luckily for us, that’s just about all we really care about.

Of course a real console application (eg: vim) cares about all terminals and you start to move away from writing control codes directly and towards using terminal apis like curses/ncurses PDQ.

For hacking purposes, VT100 terminal codes will do just fine.

I know.. what a load of historic irrelevant blather and so what and quit wasting my time! What is it good for?!

When was the last time you looked at a log file? Let me put it a different way, when did you stop looking at log files? Chances are you are watching a log file right now! I know I am!

What percentage of log output do you care about for a given task? 10%? 2%? Isn’t it a drag that it just blends in with the rest?

Of course you could grep it out, but then you lose the context… Or you might forget a term and have to reproduce the problem again (again [again]).

IMHO, a kick-aspirin use for VT100 color codes is for log highlighting, and it is some simple, it can be easily scripting into a general use script.

Here is the kind of usage I would like:

usage: cat foo | line_lite.sh ( pattern highlighting )+

pattern         is a ala grep
highlighting    is a colon separated list of colors

the colors are 

    reset        bright      dim         underline    blink
    reverse      hidden      black       red          green
    yellow       blue        magenta     cyan         white
    black_bg     red_bg      green_bg    yellow_bg    blue_bg
    magenta_bg   cyan_bg     white_bg

That way I can use it like: tail -f some.log | line_lite.sh Exception red “some other text” green:underline

In order to make this happen, I need to convert that red, green:underline crap to the appropriate VT100 code.

Natually, I do this with sed:

_line_lite_color_to_code() {
    echo ${*} | sed "s,reset,0,g;s,bright,1,g;s,dim,2,g;s,underline,4,g;s,blink,5,g;s,reverse,7,g;s,hidden,8,g;s,black,30,g;s,red,31,g;s,green,32,g;s,yellow,33,g;s,blue,34,g;s,magenta,35,g;s,cyan,36,g;s,white,37,g;s,black_bg,40,g;s,red_bg,41,g;s,green_bg,42,g;s,yellow_bg,43,g;s,blue_bg,44,g;s,magenta_bg,45,g;s,cyan_bg,46,g;s,white_bg,47,g;s,:,;,g"
}

_line_lite_code() {
    _line_lite_color_to_code ${*} | sed 's,.*,ESC[&m,'
}

This will convert “green:underline” to [32;4m

Now we just need some way to do our matching and add the color code. Once again, it’s sed time.

For each pair of pattern/color, we create a sed expression like: s!.*.*!&!g;

In this case we’d have s!.*some other text.*![32;4m&[0m!g;

All that we have to do is concatenate our sed expressions for each pattern/color pair and then call sed.

Viola! Arbitrarily colored logs!

Here is sample output from something that’s not a log:

% jar_minder_javap sun.security.provider.PolicyFile | line_lite.sh 'static'  red:bright
Compiled from "PolicyFile.java"
public class sun.security.provider.PolicyFile extends java.security.Policy{
    public sun.security.provider.PolicyFile();
    public sun.security.provider.PolicyFile(boolean);
    public void refresh();
    public boolean implies(java.security.ProtectionDomain, java.security.Permission);
    public java.security.PermissionCollection getPermissions(java.security.ProtectionDomain);
    public java.security.PermissionCollection getPermissions(java.security.CodeSource);
    protected java.security.cert.Certificate[] getSignerCertificates(java.security.CodeSource);
    static boolean access$002(sun.security.provider.PolicyFile, boolean);
    static boolean access$102(sun.security.provider.PolicyFile, boolean);
    static boolean access$202(sun.security.provider.PolicyFile, boolean);
    static boolean access$302(sun.security.provider.PolicyFile, boolean);
    static boolean access$200(sun.security.provider.PolicyFile);
    static sun.security.util.Debug access$400();
    static boolean access$500(sun.security.provider.PolicyFile, java.net.URL, sun.security.provider.PolicyInfo);
    static java.security.CodeSource access$600(sun.security.provider.PolicyFile, java.security.CodeSource, boolean);
    static java.lang.String access$700(sun.security.provider.PolicyFile, java.security.ProtectionDomain);
    static java.security.IdentityScope access$800();
    static {};
}

Sadly, my olde ascii2html script is not quite it… but I think you get the idea.

Go forth and dig them control codes, my friend! They exist only to serve your dread will.

Here is a link to a copy of the script just in case: http://brianin3d.googlepages.com/line_lite.sh


Dude has had this great link for a jillion years! http://www.fh-jena.de/~gmueller/Kurs_halle/esc_vt100.html, what a hero!

P.S.: Yes, I advocate EMACS-hateration. set -o vi or fight!

Comments (11)