dfs-datastores

dfs-datastores copied to clipboard

Can't execute any of what's show on page 68-69 of your book

1

I updated the dependencies my project.clj file, ran `lein deps`. I checked that it's in my `~/.m2/repository/` folder. I tried requiring the project in the repl in multiple ways `(require...

sb3rg

add parquet support

7

I'd like to add parquet support in addition to sequence files.

sorenmacbeth

NPE on Windows

1

A simple test file: ``` public class PailExampleTest { @SuppressWarnings({ "unchecked", "rawtypes" }) public static void main(String[] args) throws IOException { Pail pail = Pail.create("mypail"); TypedRecordOutputStream os = pail.openWrite(); os.writeObject(new...

gammay

Problem with pail consolidation on HDFS

2

I'm reading "Big Data" very interested and are trying to implement the batch layer with graph data model as master dataset. I always get a NullPointerException when I call the...

juliankeppel

I am having a problem in Lambda Architecture, Our data stored in HDFS is in fact based pail format using Thrift Serialization schemes and vertical partitioning. Is there any direct...

pankug

Here is some code to reproduce the issue ``` sh $ cd dfs-datastores/dfs-datastores/ $ lein uberjar Compiling 53 source files to /Users/bhiles/src/dfs-datastores/dfs-datastores/target/classes warning: [options] bootstrap class path not set in...

bhiles

The VersionedTap's sourceConfInit method overrides mapreduce.input.fileinputformat.inputdir in: https://github.com/nathanmarz/dfs-datastores/blob/master/dfs-datastores-cascading/src/main/java/com/backtype/cascading/tap/VersionedTap.java#L96 As a result, we can't define out of the box a cascading.tap.MultiTap as a list of VersionedTap as each VersionedTap is going...

bqm

Allow custom file formats (aside from SequenceFileFormat) to pass args

6

I'm working on an AvroFileFormat, and I'd like to be able to pass it args -- this seemed like the least bad way to make it happen. Open to suggestions...

baconz

Use the CombineFileInputFormat to avoid too many mappers

3

Change the SequenceFilePailInputFormat to use the CombineFileInputFormat. This should reduce the number of input splits for Pail sources. In my tests, several thousand splits were reduced to one. There is...

hansmire

consolidator enhancements: throw exception on delete failure, add working directory as argument

1

These are a couple of enhancements to the Consolidator. 1. At the end of the consolidator mapper I added a check on the delete of the source file and throw...

dkincaid