Ygthkb

Did 5.25" floppies undergo a change in magnetic coating?

Avoiding unpacking an array when altering its dimension

Why is working on the same position for more than 15 years not a red flag?

Why does the 31P{1H} NMR spectrum of cis-[Mo(CO)2(dppe)2] show two signals?

How to tighten battery clamp?

What is the wife of a henpecked husband called?

What's the purpose of these copper coils with resistors inside them in A Yamaha RX-V396RDS amplifier?

How to properly claim credit for peer review?

What is the difference between throw e and throw new Exception(e)?

How do ISS astronauts "get their stripes"?

Borrowing Characters

Where is the fallacy here?

What type of postprocessing gives the effect of people standing out

How to approximate rolls for potions of healing using only d6's?

Why zero tolerance on nudity in space?

Make me a metasequence

Six real numbers so that product of any five is the sixth one

How would we write a misogynistic character without offending people?

Logistics of a hovering watercraft in a fantasy setting

Use comma instead of & in table

Must a tritone substitution use a dominant seventh chord?

Called into a meeting and told we are being made redundant (laid off) and "not to share outside". Can I tell my partner?

Difference between 小吃 and 零食

Which aircraft had such a luxurious-looking navigator's station?

MongoDB merge two collections and drop documents with same value in field

MongoDB - handling embedded documents and relationsMongodb extremely slow with bulk inserts and showing erratic behaviorSeparate login for user / administrators using mongodbMongo Chunk 250000 document limit is creating problem in chunk migrationMongoDB document size for collection - impact on RAM and query performanceHow unique are MongoDB _id fields?Adding a Mongo Shard decreases the insert performanceIs it possible which documents couldn't be dumped using “mongodump”?Reliability of mongodumpMongoDB Zone Sharding without min & max key

1

In my database I have two collections, but some documents were added (possibly) at different times to both collections.

I could use mongodump and then mongorestore for merging. But then I have the same documents that were added to both collections as duplicates in my new collection. mongorestore --drop does not help neither, because the documents not necessarily have the same _id.

How to drop a document when a document with userid exists already?

mongodb mongorestore

share|improve this question

edited Nov 27 '18 at 14:50

Wuff

asked Nov 27 '18 at 14:28

WuffWuff

62

bumped to the homepage by Community♦ 11 mins ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Welcome to the StackExchange. what is MongoDB version(x,y,z)?
  
  – Md Haidar Ali Khan
  Nov 29 '18 at 9:23
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I use MongoDB v3.6.5 (build environment: distmod: debian92 distarch: x86_64 target_arch: x86_64)
  
  – Wuff
  Dec 12 '18 at 12:43
  

  
  
  
  

add a comment | 

1

In my database I have two collections, but some documents were added (possibly) at different times to both collections.

I could use mongodump and then mongorestore for merging. But then I have the same documents that were added to both collections as duplicates in my new collection. mongorestore --drop does not help neither, because the documents not necessarily have the same _id.

How to drop a document when a document with userid exists already?

mongodb mongorestore

share|improve this question

edited Nov 27 '18 at 14:50

Wuff

asked Nov 27 '18 at 14:28

WuffWuff

62

bumped to the homepage by Community♦ 11 mins ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Welcome to the StackExchange. what is MongoDB version(x,y,z)?
  
  – Md Haidar Ali Khan
  Nov 29 '18 at 9:23
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I use MongoDB v3.6.5 (build environment: distmod: debian92 distarch: x86_64 target_arch: x86_64)
  
  – Wuff
  Dec 12 '18 at 12:43
  

  
  
  
  

add a comment | 

In my database I have two collections, but some documents were added (possibly) at different times to both collections.

I could use mongodump and then mongorestore for merging. But then I have the same documents that were added to both collections as duplicates in my new collection. mongorestore --drop does not help neither, because the documents not necessarily have the same _id.

How to drop a document when a document with userid exists already?

mongodb mongorestore

share|improve this question

edited Nov 27 '18 at 14:50

Wuff

asked Nov 27 '18 at 14:28

WuffWuff

62

share|improve this question

edited Nov 27 '18 at 14:50

Wuff

asked Nov 27 '18 at 14:28

WuffWuff

62

share|improve this question

edited Nov 27 '18 at 14:50

Wuff

asked Nov 27 '18 at 14:28

WuffWuff

62

asked Nov 27 '18 at 14:28

WuffWuff

62

asked Nov 27 '18 at 14:28

WuffWuff

62

1 Answer
1

active

oldest

votes

0

There might be more elegent solutions for this, but let me answer how I merged a new collection into an original collection without importing the duplicates.

1. Compare userid fields in both collections and take the difference.


2. Use mongos $out operator to save documents to a difference collection. This new collection has all documents of the new collection, except of the duplicates.


3. 
   mongodump the difference collection and mongorestore it into the original collection.

from pymongo import MongoClient

import subprocess



# collection names

db_name = 'db'

col_original = 'col_original'

col_new = 'col_new'

field = 'userid'





# Get ids from collection

def get_ids(db, col):

    docs = db[col].aggregate([

        {'$project':

         {'_id': 0,

          'id': '$' + field}}

    ])

    docs = list(docs)

    ids = [x['id'] for x in docs]

    return ids





# Connect to MongoDB

client = MongoClient('mongodb://localhost')

db = client[db_name]



# Get difference in ids

ids_new = get_ids(db, col_new)

ids_original = get_ids(db, col_original)

ids_diff = list(set(ids_new).difference(ids_original))



# Get all documents with userid that are in col_new, but not in

# col_original. Hence, all duplicates are skipped.

db[col_new].aggregate([

    {'$match': {

        field: {'$in': ids_diff}}},

    {'$out': 'col_diff'}])



# Use mongodump to save col_diff

subprocess.check_output(['mongodump',

                         '-d',

                         db_name,

                         '-c',

                         'col_diff'])



# Merge col_diff into col_original

subprocess.check_output(['mongorestore',

                         '-d',

                         db_name,

                         '-c',

                         col_original,

                         'dump/' + db_name + '/col_diff.bson'])

share|improve this answer

answered Dec 12 '18 at 12:41

WuffWuff

62

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f223551%2fmongodb-merge-two-collections-and-drop-documents-with-same-value-in-field%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

0

There might be more elegent solutions for this, but let me answer how I merged a new collection into an original collection without importing the duplicates.

1. Compare userid fields in both collections and take the difference.


2. Use mongos $out operator to save documents to a difference collection. This new collection has all documents of the new collection, except of the duplicates.


3. 
   mongodump the difference collection and mongorestore it into the original collection.

from pymongo import MongoClient

import subprocess



# collection names

db_name = 'db'

col_original = 'col_original'

col_new = 'col_new'

field = 'userid'





# Get ids from collection

def get_ids(db, col):

    docs = db[col].aggregate([

        {'$project':

         {'_id': 0,

          'id': '$' + field}}

    ])

    docs = list(docs)

    ids = [x['id'] for x in docs]

    return ids





# Connect to MongoDB

client = MongoClient('mongodb://localhost')

db = client[db_name]



# Get difference in ids

ids_new = get_ids(db, col_new)

ids_original = get_ids(db, col_original)

ids_diff = list(set(ids_new).difference(ids_original))



# Get all documents with userid that are in col_new, but not in

# col_original. Hence, all duplicates are skipped.

db[col_new].aggregate([

    {'$match': {

        field: {'$in': ids_diff}}},

    {'$out': 'col_diff'}])



# Use mongodump to save col_diff

subprocess.check_output(['mongodump',

                         '-d',

                         db_name,

                         '-c',

                         'col_diff'])



# Merge col_diff into col_original

subprocess.check_output(['mongorestore',

                         '-d',

                         db_name,

                         '-c',

                         col_original,

                         'dump/' + db_name + '/col_diff.bson'])

share|improve this answer

answered Dec 12 '18 at 12:41

WuffWuff

62

add a comment | 

0

There might be more elegent solutions for this, but let me answer how I merged a new collection into an original collection without importing the duplicates.

1. Compare userid fields in both collections and take the difference.


2. Use mongos $out operator to save documents to a difference collection. This new collection has all documents of the new collection, except of the duplicates.


3. 
   mongodump the difference collection and mongorestore it into the original collection.

from pymongo import MongoClient

import subprocess



# collection names

db_name = 'db'

col_original = 'col_original'

col_new = 'col_new'

field = 'userid'





# Get ids from collection

def get_ids(db, col):

    docs = db[col].aggregate([

        {'$project':

         {'_id': 0,

          'id': '$' + field}}

    ])

    docs = list(docs)

    ids = [x['id'] for x in docs]

    return ids





# Connect to MongoDB

client = MongoClient('mongodb://localhost')

db = client[db_name]



# Get difference in ids

ids_new = get_ids(db, col_new)

ids_original = get_ids(db, col_original)

ids_diff = list(set(ids_new).difference(ids_original))



# Get all documents with userid that are in col_new, but not in

# col_original. Hence, all duplicates are skipped.

db[col_new].aggregate([

    {'$match': {

        field: {'$in': ids_diff}}},

    {'$out': 'col_diff'}])



# Use mongodump to save col_diff

subprocess.check_output(['mongodump',

                         '-d',

                         db_name,

                         '-c',

                         'col_diff'])



# Merge col_diff into col_original

subprocess.check_output(['mongorestore',

                         '-d',

                         db_name,

                         '-c',

                         col_original,

                         'dump/' + db_name + '/col_diff.bson'])

share|improve this answer

answered Dec 12 '18 at 12:41

WuffWuff

62

add a comment | 

There might be more elegent solutions for this, but let me answer how I merged a new collection into an original collection without importing the duplicates.

1. Compare userid fields in both collections and take the difference.


2. Use mongos $out operator to save documents to a difference collection. This new collection has all documents of the new collection, except of the duplicates.


3. 
   mongodump the difference collection and mongorestore it into the original collection.

from pymongo import MongoClient

import subprocess



# collection names

db_name = 'db'

col_original = 'col_original'

col_new = 'col_new'

field = 'userid'





# Get ids from collection

def get_ids(db, col):

    docs = db[col].aggregate([

        {'$project':

         {'_id': 0,

          'id': '$' + field}}

    ])

    docs = list(docs)

    ids = [x['id'] for x in docs]

    return ids





# Connect to MongoDB

client = MongoClient('mongodb://localhost')

db = client[db_name]



# Get difference in ids

ids_new = get_ids(db, col_new)

ids_original = get_ids(db, col_original)

ids_diff = list(set(ids_new).difference(ids_original))



# Get all documents with userid that are in col_new, but not in

# col_original. Hence, all duplicates are skipped.

db[col_new].aggregate([

    {'$match': {

        field: {'$in': ids_diff}}},

    {'$out': 'col_diff'}])



# Use mongodump to save col_diff

subprocess.check_output(['mongodump',

                         '-d',

                         db_name,

                         '-c',

                         'col_diff'])



# Merge col_diff into col_original

subprocess.check_output(['mongorestore',

                         '-d',

                         db_name,

                         '-c',

                         col_original,

                         'dump/' + db_name + '/col_diff.bson'])

share|improve this answer

answered Dec 12 '18 at 12:41

WuffWuff

62

from pymongo import MongoClient

import subprocess



# collection names

db_name = 'db'

col_original = 'col_original'

col_new = 'col_new'

field = 'userid'





# Get ids from collection

def get_ids(db, col):

    docs = db[col].aggregate([

        {'$project':

         {'_id': 0,

          'id': '$' + field}}

    ])

    docs = list(docs)

    ids = [x['id'] for x in docs]

    return ids





# Connect to MongoDB

client = MongoClient('mongodb://localhost')

db = client[db_name]



# Get difference in ids

ids_new = get_ids(db, col_new)

ids_original = get_ids(db, col_original)

ids_diff = list(set(ids_new).difference(ids_original))



# Get all documents with userid that are in col_new, but not in

# col_original. Hence, all duplicates are skipped.

db[col_new].aggregate([

    {'$match': {

        field: {'$in': ids_diff}}},

    {'$out': 'col_diff'}])



# Use mongodump to save col_diff

subprocess.check_output(['mongodump',

                         '-d',

                         db_name,

                         '-c',

                         'col_diff'])



# Merge col_diff into col_original

subprocess.check_output(['mongorestore',

                         '-d',

                         db_name,

                         '-c',

                         col_original,

                         'dump/' + db_name + '/col_diff.bson'])

share|improve this answer

answered Dec 12 '18 at 12:41

WuffWuff

62

answered Dec 12 '18 at 12:41

WuffWuff

62

answered Dec 12 '18 at 12:41

WuffWuff

62