[FIX] Venn Diagram is slow for big datasets

[AndrejaKovacic]

Issue

Fixes #3989

Description of changes

Refactored venn over rows.

TODO:

• Output duplicates

Includes

• Code changes
• Tests

[codecov]

Codecov Report

Merging #4400 into master will increase coverage by 0.24%.
The diff coverage is 95.96%.

@@            Coverage Diff             @@
##           master    #4400      +/-   ##
==========================================
+ Coverage   87.19%   87.43%   +0.24%     
==========================================
  Files         401      405       +4     
  Lines       72938    73990    +1052     
==========================================
+ Hits        63595    64692    +1097     
+ Misses       9343     9298      -45

[janezd]

This is quite a work. Nice tricks in the code, too.

I started commenting, but then found it easier to do the changes myself (they are just, minor, stylistic) and comment on them. Please review my changes and reject them if any of them are wrong.

I spent some time on trying to follow and understand the code, but I'll have to invest some more time -- or let you explain it to me on Friday.

[janezd]

Run the widget (from main). Select a few regions. In "Rows (instances)" change the combo to some attribute. It crashes. If it doesn't switch to the other attribute. It crashes then.

The error occurs a[mask] = values: ValueError: shape mismatch: value array of shape (27,) could not be broadcast to indexing result of shape (27,1).

[AndrejaKovacic]

I think reshaping should do the trick. I also removed relevant_keys parameter, since indices already contain that information in create_from_rows.

[janezd]

Run widget from main. Use rowwise, by instance. Click Output duplicates and select some area. Widget crashes with

  File "/Users/janez/orange3/Orange/widgets/visualize/owvenndiagram.py", line 663, in commit
    selected = self.create_from_rows(selected_ids, False)
  File "/Users/janez/orange3/Orange/widgets/visualize/owvenndiagram.py", line 598, in create_from_rows
    return self.extract_rowwise_duplicates(var_dict, relevant_ids)
  File "/Users/janez/orange3/Orange/widgets/visualize/owvenndiagram.py", line 634, in extract_rowwise_duplicates
    x, m, y, t_ids = self.expand_table(extracted, all_atrs, all_metas, all_cv)
  File "/Users/janez/orange3/Orange/widgets/visualize/owvenndiagram.py", line 616, in expand_table
    array[:, perm] = b
ValueError: could not convert string to float: 'YGL048C'

[janezd]

Run widget from main. Use rowwise. Match by "Test". Check Output duplicates and select some (non-empry) area. Widget crashes with

  File "/Users/janez/orange3/Orange/widgets/visualize/owvenndiagram.py", line 663, in commit
    selected = self.create_from_rows(selected_ids, False)
  File "/Users/janez/orange3/Orange/widgets/visualize/owvenndiagram.py", line 598, in create_from_rows
    return self.extract_rowwise_duplicates(var_dict, relevant_ids)
  File "/Users/janez/orange3/Orange/widgets/visualize/owvenndiagram.py", line 640, in extract_rowwise_duplicates
    values = {'attributes': [np.vstack(all_x)],
  File "/Users/janez/miniconda3/envs/o3/lib/python3.7/site-packages/numpy/core/shape_base.py", line 283, in vstack
    return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
ValueError: need at least one array to concatenate

[AndrejaKovacic]

Run widget from main. Use rowwise. Match by "Test". Check Output duplicates and select some (non-empry) area. Widget crashes with

  File "/Users/janez/orange3/Orange/widgets/visualize/owvenndiagram.py", line 663, in commit
    selected = self.create_from_rows(selected_ids, False)
  File "/Users/janez/orange3/Orange/widgets/visualize/owvenndiagram.py", line 598, in create_from_rows
    return self.extract_rowwise_duplicates(var_dict, relevant_ids)
  File "/Users/janez/orange3/Orange/widgets/visualize/owvenndiagram.py", line 640, in extract_rowwise_duplicates
    values = {'attributes': [np.vstack(all_x)],
  File "/Users/janez/miniconda3/envs/o3/lib/python3.7/site-packages/numpy/core/shape_base.py", line 283, in vstack
    return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
ValueError: need at least one array to concatenate

This crash was caused by Test being a StringVariable only by name, but not by nature :)

[janezd]

This crash was caused by Test being a StringVariable only by name, but not by nature :)

I see. It's heart wasn't in it. :)