Support built-in bulk operations

[michaelklishin]

See http://docs.mongodb.org/manual/core/bulk-write-operations/ and https://groups.google.com/forum/#!forum/clojure-mongodb.

[vesse]

Is there a workaround available meanwhile? Would need to run initializeUnorderedBulkOp, add operations to the returned bulk, and then execute it.

[michaelklishin]

Monger accepts DB instances, so you can always use Java interop and monger.conversion.

[alolis]

@michaelklishin, I am using monger 1.8.0 with Mongo 2.6.5 and I tried to follow your work around until this is properly implemented but I am getting a BulkWriteException on duplicate key, although I shouldn't since I am using unordered writes.

(monger.core/connect! my-connection-map)
(monger.core/use-db! my-database-string)
(def db (monger.core/get-db))
(def collection (.getCollection db "items"))
(def bulk-write (.initializeUnorderedBulkOperation collection))

(.insert bulk-write (monger.conversion/to-db-object {:_id "a" :foo "bar"}))
(.insert bulk-write (monger.conversion/to-db-object {:_id "a" :foo "bar"})) ;; Dup key 
(.insert bulk-write (monger.conversion/to-db-object {:_id "b" :foo "bar"}))

(.execute bulk-write) ;; BulkWriteException here. First document gets inserted in database, third one does not, although it should.

I did have a look inside monger.core source but I couldn't find a problem. I also tried the bulk operation via the Mongo shell and it worked as expected so I am thinking it might be something in the mongo java driver (maybe just an update will do?) or in the way you are initializing the java driver (although I didn't see anything weird). You will probably have a better idea than I do at the moment.

[michaelklishin]

Why do you expect bulk writes with duplicate keys to work? I'm not sure how they can, regardless of the client or database.

[alolis]

The documentation states that this is possible (and indeed it is because I tried it in the mongo shell)

http://docs.mongodb.org/manual/reference/method/db.collection.initializeUnorderedBulkOp/

Error Handling
If an error occurs during the processing of one of the write operations, MongoDB will continue to process remaining write operations in the list.

[vesse]

Here's a simple upsert example just in case someone's interested (since it didn't go as I expected from the shell examples):

(let [col  (.getCollection db "collection_name")
      bulk (.initializeUnorderedBulkOperation col)]
  (.update (.upsert (.find bulk (monger.conversion/to-db-object {:_id (:_id object)})))
           (monger.conversion/to-db-object {$set (dissoc object :_id)}))
  (.execute bulk))

[alolis]

Well, it seems that the problem I was describing is in the mongo-java-driver version (v2.12.1) which monger 1.8.0 is using.

https://jira.mongodb.org/browse/JAVA-1541

EDIT: I confirm that monger v1.8.0 with mongo-java-driver v2.13.0-rc0 works as expected.

[michaelklishin]

@alolis please submit a PR that upgrades the dependency.

[alolis]

@michaelklishin , sorry for the delay. Yeah sure, I can do it first thing tomorrow morning but please verify the commit in order to be 100% sure.

Is this the one? https://github.com/michaelklishin/monger/tree/v1.8.0

[sql1]

I have also good in using Mongodb and I like all functionality. Here I would like to mention a link that Will help you. All the things is discussed properly. Feel free to visit

[robert-m-johnson]

I'd be interested in having a look at this. Did you have anything in mind for the API?

[michaelklishin]

@robert-m-johnson thank you. The API can vary from passing in a sequence/collection to a full-blown mini-DSL à la what Neocons and Elastisch have. I'm open to suggestions and would need to take a look at what the modern Java driver provides.

[robert-m-johnson]

The newer versions of the driver have you build up a sequence of WriteModel objects, like the following example taken from http://stackoverflow.com/questions/31470702/

MongoCollection<Document> collection = db.getCollection("sample");

List<WriteModel<Document>> updates = Arrays.<WriteModel<Document>>asList(
    new UpdateOneModel<Document>(
        new Document(),                   // find part
        new Document("$set",1),           // update part
        new UpdateOptions().upsert(true)  // options like upsert
    )
);

BulkWriteResult bulkWriteResult = collection.bulkWrite(updates);

It makes the most sense to me to go with passing in a sequence since it would both be idiomatic Clojure and similar to the Java API, but I guess this could be augmented with a mini-DSL if that wouldn't be too confusing.

My questions are

1. How do we represent the type of the type of each individual write operation?
2. How do we indicate whether the bulk write is ordered or not?

My initial thoughts on these are to simply use a symbol to represent a type, and a function argument could indicate ordered or unordered. A vector would represent each write operation. So the Monger version of the above Java example could look something like

(mc/bulk-write db "sample" [[:update-one {} {"$set" 1} {:upsert true}]] :ordered)

[mcferren]

Would be very grateful for this mc/bulk-write feature

[viebel]

Any progress on adding bulk-write to monger?