MongoDB Connector

To add Mongodb Almaren dependency to your sbt build:

libraryDependencies += "com.github.music-of-the-ainur" %% "mongodb-almaren" % "0.0.9-3.4"
libraryDependencies += "org.mongodb.spark" % "mongo-spark-connector" % "10.2.0"

To run in spark-shell:

spark-shell --master local[*] --packages "com.github.music-of-the-ainur:almaren-framework_2.12:0.9.10-3.4,com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-3.4,org.mongodb.spark:mongo-spark-connector:10.2.0"

MongoDB Connector is available in Maven Central repository.

version	Connector Artifact
Spark 3.4.x and scala 2.13	com.github.music-of-the-ainur:mongodb-almaren_2.13:0.0.9-3.4
Spark 3.4.x and scala 2.12	com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-3.3
Spark 3.3.x and scala 2.13	com.github.music-of-the-ainur:mongodb-almaren_2.13:0.0.9-3.3
Spark 3.3.x and scala 2.12	com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-3.3
Spark 3.2.x and scala 2.12	com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-3.2
Spark 3.1.x and scala 2.12	com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-3.1
Spark 2.4.x and scala 2.12	com.github.music-of-the-ainur:mongodb-almaren_2.12:0.0.9-2.4
Spark 2.4.x and scala 2.11	com.github.music-of-the-ainur:mongodb-almaren_2.11:0.0.9-2.4

Source and Target

Connector was implemented using: https://github.com/mongodb/mongo-spark.

Example

Source

Parameters:

Parameters	Description
hosts	localhost:27017
database	foo
collection	bar
user	username
password	password
stringPrefix	this is used to specify MongoDb Connection type(srv) https://docs.mongodb.com/manual/reference/connection-string/
options	extra connector options

For Standalone Connection Type Mongo

import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit

almaren.builder.sourceMongoDb("localhost","foo","bar")

For srv Connection Type Mongo

import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit

almaren.builder.sourceMongoDb("localhost","foo","bar",None,None,Some("srv"))

Parameters for Uri:

Parameters	Description
uri	mongodb://localhost:27017/foo
collection	bar
options	extra connector options

For Connection Uri Type Mongo

import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit

almaren.builder.sourceMongoDbUri("mongodb://localhost:27017/foo","bar")

Target

Parameters:

Parameters	Description
hosts	localhost:27017
database	foo
collection	bar
user	username
password	password
stringPrefix	this is used to specify MongoDb Connection type(srv) https://docs.mongodb.com/manual/reference/connection-string/
options	extra connector options
saveMode	SaveMode.Overwrite

For Standalone Connection Type Mongo

import org.apache.spark.sql.SaveMode
import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit

almaren.builder.targetMongoDb("localhost","foo","bar", saveMode = SaveMode.Overwrite)

For srv Connection Type Mongo

import org.apache.spark.sql.SaveMode
import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit

almaren.builder.targetMongoDb("localhost","foo","bar",None,None,Some("srv"), saveMode = SaveMode.Overwrite)

Parameters for Uri:

Parameters	Description
uri	mongodb://localhost:27017/foo
collection	bar
options	extra connector options
saveMode	SaveMode.Overwrite

For Connection Uri Type Mongo

import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.mongodb.MongoDb.MongoImplicit

almaren.builder.targetMongoDbUri("mongodb://localhost:27017/foo","bar", saveMode = SaveMode.Overwrite)