dfs-datastores

dfs-datastores copied to clipboard

Check type of object being written via TypedRecordOutputStream.writeObject(obj); Override Pail.toString() for nicer printing

1

TypedRecordOutputStream.writeObject(obj); should check type of object being written to Pail. Pail.toString() is also overriden to print Pail info more nicely (not just for this issue of course). Result is following...

vmarcinko

NPE in VersionedTap.sourceConfInit

6

When I create a new versioned tap (using VersionedKeyValSource from Scalding) I get an NPE: Caused by: java.lang.NullPointerException at org.apache.hadoop.mapred.FileInputFormat.getPathStrings(FileInputFormat.java:342) at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:288) at com.backtype.cascading.tap.VersionedTap.sourceConfInit(VersionedTap.java:88) at com.backtype.cascading.tap.VersionedTap.sourceConfInit(VersionedTap.java:19) at cascading.flow.hadoop.HadoopFlowStep.initFromSources(HadoopFlowStep.java:332) at cascading.flow.hadoop.HadoopFlowStep.getInitializedConfig(HadoopFlowStep.java:99)...

mikegagnon

PailStructure passed to Pail.create is not used in the output stream

2

I have a slightly modified implementation of PailStructure where i store some state (say myvar) in the Implemented Object which is used for Ser/De. So my code looks something like...

kul

Right now, a PailRecordWriter can open an unlimited number of files. Instead of using a HashMap to contain the mapping of attributes to open files, PailRecordWriter should use a LinkedHashMap...

derrickburns

@johnynek, moved your issue over to here. Exception in thread "flow Tutorial6" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: manhattansink:kv.test:LATEST at org.apache.hadoop.fs.Path.initialize(Path.java:148) at org.apache.hadoop.fs.Path.(Path.java:126) at cascading.tap.hadoop.util.Hadoop18TapUtil.cleanupTapMetaData(Hadoop18TapUtil.java:185) at cascading.flow.hadoop.HadoopFlowStep.cleanTapMetaData(HadoopFlowStep.java:272) at cascading.flow.hadoop.HadoopFlowStep.clean(HadoopFlowStep.java:257)...

sritchie

Snapshot doesn't allow injection of a FileSystem to use for the snapshot destination

2

The Pail.create() methods allow a FileSystem to be used in the creation of the pail. However, when creating a snapshot of a pail the only option is to include the...

dkincaid

Snapshot fails if process can't write to /tmp

1

Since "/tmp/filecopy" is hardcoded in FileCopyInputFormat as tmproot if a snapshot is run and the user running the process isn't able to write to "/tmp" or "/tmp/filecopy" the snapshot will...

dkincaid

When PailTap is used, sink taps do not use randomized names as expected. Instead, they use the default hadoop naming scheme. expected: ``` target/randomized.pailfile ``` actual: ``` target/part-00000.pailfile ``` I've...

fkautz

Usable compression codecs are currently determined at compile time. Allow the user to inject additional codecs programmatically without the need for dfs-datastores to know about them at compile time. Also,...

fkautz

The default SkipStrategy in Cascading 2.1 and greater uses the timestamp of the directory returned by getPath(), which in the case of a Pail doesn't change. We need to override...

sorenmacbeth