Ygthkb

Number of FLOP (Floating Point Operations) for exponentiation

Can we use the stored gravitational potential energy of a building to produce power?

What do you call a fact that doesn't match the settings?

The vanishing of sum of coefficients: symmetric polynomials

Using loops to create tables

How to define new environment, using proof-environment

What are the advantages of using `make` for small projects?

Using only 1s, make 29 with the minimum number of digits

Manipulating a general length function

Reference on complex cobordism

Overfitting and Underfitting

Issues with new Macs: hardware makes them difficult to use … what options might be available in the future?

Why avoid shared user accounts?

Slow moving projectiles from a hand-held weapon - how do they reach the target?

How to generate a matrix with certain conditions

Can you earn endless XP using a Flameskull and its self-revival feature?

How experienced do I need to be to go on a photography workshop?

How did the original light saber work?

En Passant For Beginners

Why is button three on trumpet almost never used alone?

It took me a lot of time to make this, pls like. (YouTube Comments #1)

What is better: yes / no radio, or simple checkbox?

How can I improve my fireworks photography?

Does Windows 10's telemetry include sending *.doc files if Word crashed?

Is there a way to drop duplicated rows based on an unhashable column?

how to impute missing values on numpy array created by train_test_split from pandas.DataFrame?How do I convert a pandas dataframe to a 1d array?How to change a cell in Pandas dataframe with respective frequency of the cell in respective columnPopulate column based on previous row with a twistCounting the occurrence of each string in a pandas dataframe columnCreate new data frames from existing data frame based on unique column valuesHow to statistically prove that a column in a dataframe is not neededShould I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column?How can I merge 2+ DataFrame objects without duplicating column names?operations on multiple entries in one column based on conditions meet from multiple column entries

1

$begingroup$

i have a pandas dataframe df with one column z filled with set values

i want to drop duplicated rows where 2 rows are considered duplicated version of one another when they have same column z values ( which are sets ).

import pandas as pd



lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) , ( 'b' , 'a' , { 'a' , 'b' } ) ]

lbls = [ 'x' , 'y' , 'z' ]

df = pd.DataFrame.from_records( lnks , columns = lbls )

Trying to drop duplicated rows based on column z values :

df.drop_duplicates( subset = 'z' , keep='first')

And i get the error message :

TypeError: unhashable type: 'set'

Is there a way to drop duplicated rows based on a unhashable typed column ?

python pandas dataframe

share|improve this question

edited 1 hour ago

n1k31t4

6,1362319

asked 5 hours ago

Fabrice BOUCHARELFabrice BOUCHAREL

585

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I assume it is a typo - but there isn't actually a duplicate in row z anyway because one b also has a space: 'b '.
  $endgroup$
  – n1k31t4
  4 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  right. I've made a correction. thx.
  $endgroup$
  – Fabrice BOUCHAREL
  3 hours ago
  

  
  
  
  

add a comment | 

1

$begingroup$

i have a pandas dataframe df with one column z filled with set values

i want to drop duplicated rows where 2 rows are considered duplicated version of one another when they have same column z values ( which are sets ).

import pandas as pd



lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) , ( 'b' , 'a' , { 'a' , 'b' } ) ]

lbls = [ 'x' , 'y' , 'z' ]

df = pd.DataFrame.from_records( lnks , columns = lbls )

Trying to drop duplicated rows based on column z values :

df.drop_duplicates( subset = 'z' , keep='first')

And i get the error message :

TypeError: unhashable type: 'set'

Is there a way to drop duplicated rows based on a unhashable typed column ?

python pandas dataframe

share|improve this question

edited 1 hour ago

n1k31t4

6,1362319

asked 5 hours ago

Fabrice BOUCHARELFabrice BOUCHAREL

585

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I assume it is a typo - but there isn't actually a duplicate in row z anyway because one b also has a space: 'b '.
  $endgroup$
  – n1k31t4
  4 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  right. I've made a correction. thx.
  $endgroup$
  – Fabrice BOUCHAREL
  3 hours ago
  

  
  
  
  

add a comment | 

$begingroup$

i have a pandas dataframe df with one column z filled with set values

i want to drop duplicated rows where 2 rows are considered duplicated version of one another when they have same column z values ( which are sets ).

import pandas as pd



lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) , ( 'b' , 'a' , { 'a' , 'b' } ) ]

lbls = [ 'x' , 'y' , 'z' ]

df = pd.DataFrame.from_records( lnks , columns = lbls )

Trying to drop duplicated rows based on column z values :

df.drop_duplicates( subset = 'z' , keep='first')

And i get the error message :

TypeError: unhashable type: 'set'

Is there a way to drop duplicated rows based on a unhashable typed column ?

python pandas dataframe

share|improve this question

edited 1 hour ago

n1k31t4

6,1362319

asked 5 hours ago

Fabrice BOUCHARELFabrice BOUCHAREL

585

$endgroup$

import pandas as pd



lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) , ( 'b' , 'a' , { 'a' , 'b' } ) ]

lbls = [ 'x' , 'y' , 'z' ]

df = pd.DataFrame.from_records( lnks , columns = lbls )

df.drop_duplicates( subset = 'z' , keep='first')

TypeError: unhashable type: 'set'

share|improve this question

edited 1 hour ago

n1k31t4

6,1362319

asked 5 hours ago

Fabrice BOUCHARELFabrice BOUCHAREL

585

share|improve this question

edited 1 hour ago

n1k31t4

6,1362319

asked 5 hours ago

Fabrice BOUCHARELFabrice BOUCHAREL

585

edited 1 hour ago

n1k31t4

6,1362319

edited 1 hour ago

n1k31t4

6,1362319

asked 5 hours ago

Fabrice BOUCHARELFabrice BOUCHAREL

585

asked 5 hours ago

Fabrice BOUCHARELFabrice BOUCHAREL

585

1 Answer
1

active

oldest

votes

2

$begingroup$

It is true that a set is not hashable (it cannot be used as a key in a hashmap a.k.a a dictionary). So what you can do is to just convert the column to a type that is hashable - I would go for a tuple.

I made a new column that is just the "z" column you had, converted to tuples. Then you can use the same method you tried to, on the new column:

In [1] : import pandas as pd 

    ...:  

    ...: lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) 

    ...: , ( 'b' , 'a' , { 'a' , 'b' } ) ] 

    ...: lbls = [ 'x' , 'y' , 'z' ] 

    ...: df = pd.DataFrame.from_records( lnks , columns = lbls)                 



In [2]: df["z_tuple"] = df.z.apply(lambda x: tuple(x))                         



In [3]: df.drop_duplicates(subset="z_tuple", keep="first")                     

Out[3]: 

   x  y       z z_tuple

0  a  b  {b, a}  (b, a)

1  b  c  {c, b}  (c, b)

The apply method lets you apply a function to each item in a column, and then returns the values as a new column (a Pandas Series object). This lets you assign it back to the original DataFrame as a new column, as I did.

You can also remove the "z_tuple" column then if you no longer want it:

In [4] : df.drop("z_tuple", axis=1, inplace=True)                               



In [5] : df                                                                     

Out[5] : 

   x  y       z

0  a  b  {b, a}

1  b  c  {c, b}

2  b  a  {b, a}

share|improve this answer

answered 4 hours ago

n1k31t4n1k31t4

6,1362319

$endgroup$

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46541%2fis-there-a-way-to-drop-duplicated-rows-based-on-an-unhashable-column%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

2

$begingroup$

It is true that a set is not hashable (it cannot be used as a key in a hashmap a.k.a a dictionary). So what you can do is to just convert the column to a type that is hashable - I would go for a tuple.

I made a new column that is just the "z" column you had, converted to tuples. Then you can use the same method you tried to, on the new column:

In [1] : import pandas as pd 

    ...:  

    ...: lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) 

    ...: , ( 'b' , 'a' , { 'a' , 'b' } ) ] 

    ...: lbls = [ 'x' , 'y' , 'z' ] 

    ...: df = pd.DataFrame.from_records( lnks , columns = lbls)                 



In [2]: df["z_tuple"] = df.z.apply(lambda x: tuple(x))                         



In [3]: df.drop_duplicates(subset="z_tuple", keep="first")                     

Out[3]: 

   x  y       z z_tuple

0  a  b  {b, a}  (b, a)

1  b  c  {c, b}  (c, b)

The apply method lets you apply a function to each item in a column, and then returns the values as a new column (a Pandas Series object). This lets you assign it back to the original DataFrame as a new column, as I did.

You can also remove the "z_tuple" column then if you no longer want it:

In [4] : df.drop("z_tuple", axis=1, inplace=True)                               



In [5] : df                                                                     

Out[5] : 

   x  y       z

0  a  b  {b, a}

1  b  c  {c, b}

2  b  a  {b, a}

share|improve this answer

answered 4 hours ago

n1k31t4n1k31t4

6,1362319

$endgroup$

add a comment | 

2

$begingroup$

It is true that a set is not hashable (it cannot be used as a key in a hashmap a.k.a a dictionary). So what you can do is to just convert the column to a type that is hashable - I would go for a tuple.

I made a new column that is just the "z" column you had, converted to tuples. Then you can use the same method you tried to, on the new column:

In [1] : import pandas as pd 

    ...:  

    ...: lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) 

    ...: , ( 'b' , 'a' , { 'a' , 'b' } ) ] 

    ...: lbls = [ 'x' , 'y' , 'z' ] 

    ...: df = pd.DataFrame.from_records( lnks , columns = lbls)                 



In [2]: df["z_tuple"] = df.z.apply(lambda x: tuple(x))                         



In [3]: df.drop_duplicates(subset="z_tuple", keep="first")                     

Out[3]: 

   x  y       z z_tuple

0  a  b  {b, a}  (b, a)

1  b  c  {c, b}  (c, b)

The apply method lets you apply a function to each item in a column, and then returns the values as a new column (a Pandas Series object). This lets you assign it back to the original DataFrame as a new column, as I did.

You can also remove the "z_tuple" column then if you no longer want it:

In [4] : df.drop("z_tuple", axis=1, inplace=True)                               



In [5] : df                                                                     

Out[5] : 

   x  y       z

0  a  b  {b, a}

1  b  c  {c, b}

2  b  a  {b, a}

share|improve this answer

answered 4 hours ago

n1k31t4n1k31t4

6,1362319

$endgroup$

add a comment | 

$begingroup$

It is true that a set is not hashable (it cannot be used as a key in a hashmap a.k.a a dictionary). So what you can do is to just convert the column to a type that is hashable - I would go for a tuple.

I made a new column that is just the "z" column you had, converted to tuples. Then you can use the same method you tried to, on the new column:

In [1] : import pandas as pd 

    ...:  

    ...: lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) 

    ...: , ( 'b' , 'a' , { 'a' , 'b' } ) ] 

    ...: lbls = [ 'x' , 'y' , 'z' ] 

    ...: df = pd.DataFrame.from_records( lnks , columns = lbls)                 



In [2]: df["z_tuple"] = df.z.apply(lambda x: tuple(x))                         



In [3]: df.drop_duplicates(subset="z_tuple", keep="first")                     

Out[3]: 

   x  y       z z_tuple

0  a  b  {b, a}  (b, a)

1  b  c  {c, b}  (c, b)

The apply method lets you apply a function to each item in a column, and then returns the values as a new column (a Pandas Series object). This lets you assign it back to the original DataFrame as a new column, as I did.

You can also remove the "z_tuple" column then if you no longer want it:

In [4] : df.drop("z_tuple", axis=1, inplace=True)                               



In [5] : df                                                                     

Out[5] : 

   x  y       z

0  a  b  {b, a}

1  b  c  {c, b}

2  b  a  {b, a}

share|improve this answer

answered 4 hours ago

n1k31t4n1k31t4

6,1362319

$endgroup$

In [1] : import pandas as pd 

    ...:  

    ...: lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) 

    ...: , ( 'b' , 'a' , { 'a' , 'b' } ) ] 

    ...: lbls = [ 'x' , 'y' , 'z' ] 

    ...: df = pd.DataFrame.from_records( lnks , columns = lbls)                 



In [2]: df["z_tuple"] = df.z.apply(lambda x: tuple(x))                         



In [3]: df.drop_duplicates(subset="z_tuple", keep="first")                     

Out[3]: 

   x  y       z z_tuple

0  a  b  {b, a}  (b, a)

1  b  c  {c, b}  (c, b)

In [4] : df.drop("z_tuple", axis=1, inplace=True)                               



In [5] : df                                                                     

Out[5] : 

   x  y       z

0  a  b  {b, a}

1  b  c  {c, b}

2  b  a  {b, a}

share|improve this answer

answered 4 hours ago

n1k31t4n1k31t4

6,1362319

answered 4 hours ago

n1k31t4n1k31t4

6,1362319

answered 4 hours ago

n1k31t4n1k31t4

6,1362319