Add wu-* commands for filesystem, etc...

[dieterichlawson]

1. Add Gorillib::Runner.run command with childprocess gem

2. Add job commands:

• wu-ps
• wu-kill

3. Add filesystem commands:

• wu-dump
• wu-lign
• wu-ls
• wu-du
• wu-cp
• wu-mv
• wu-rm
• wu-mkdir
• wu-put
• wu-get

[dhruvbansal]

Many of these proposed wu- binaries seem like unnecessary aliases for already simple commands. wu-mv foo bar isn't so much better than hadoop fs -mv foo bar that it justifies a brand new executable on my $PATH, IMO. Or is the idea that we are "making it easier" or "providing an abstraction above Hadoop" ... ?

Stuff like wu-du and wu-lign (of course) are more useful but wu-du clearly belongs in wukong-hadoop.

Can we think about this list a little more critically is what I'm saying, I suppose.

[mrflip]

Yes --

• these will currently offer pathname helpers that are extremely useful as your projects grow beyond one-or-two scripts.
• they present a more familiar interface -- wu-rm -r, not hadoop fs -rmr -- and a unified experience, with configliere/gnu style options and help (--skip_trash, not -skipTrash)
• they will let us swap in a thrift back end that is much more performant -- no more waiting 1000ms for the JVM to start.
• they will later become file system and datastore agnostic: real fs vs hdfs, and local vs Infochimps platform; and maybe even into the data store as appropriate.

The Hadoop file system commands are a true evil, and we've found these aliases really help when doing large project development. You already had most of them in your $PATH already as hdp-*.

No strong opinion on wukong vs wukong-hadoop, though they will eventually become non-hadoop-only.

Flip

Sent from my iPad

On Feb 20, 2013, at 12:28 AM, Dhruv Bansal [email protected] wrote:

Many of these proposed wu- binaries seem like unnecessary aliases for already simple commands. wu-mv foo bar isn't so much better than hadoop fs -mv foo bar that it justifies a brand new executable on my $PATH, IMO. Or is the idea that we are "making it easier" or "providing an abstraction above Hadoop" ... ?

Stuff like wu-du and wu-lign (of course) are more useful but wu-du clearly belongs in wukong-hadoop.

Can we think about this list a little more critically is what I'm saying, I suppose.

—
Reply to this email directly or view it on GitHub.