Merge branch 'twitter:main' into main

twitter · Sep 29, 2024 · c2ae4cd · c2ae4cd
2 parents 8e33114 + 50526b6
commit c2ae4cd
Show file tree

Hide file tree

Showing 48 changed files with 6,028 additions and 1,505 deletions.
diff --git a/README.md b/README.md
@@ -65,7 +65,7 @@ $ cd sourcecode
 $ python main.py
 ```
 
-Most versions of Python3 should work, but we have tested the code with Python 3.8.
+Multiple versions of Python3 should work, but we have tested the code with Python 3.10.
 
 ### Community Notes data
 

diff --git a/documentation/about/faq.md b/documentation/about/faq.md
@@ -47,7 +47,9 @@ Post authors with Community Notes on their posts can request additional review o
 {% /accordionItem %}
 
 {% accordionItem title="How do I opt-out of the program?" %}
-If you join the program but later want to leave, DM @CommunityNotes to let the team know.
+If you join the program but later want to leave, you can do so in your [Community Notes settings](https://x.com/i/communitynotes/notification_settings).
+
+If you leave and then later want to re-join, you can do so on your [Community Notes alias profile](http://x.com/i/communitynotes/u/me).
 {% /accordionItem %}
 {% /accordionSection %}
 

diff --git a/documentation/contributing/notifications.md b/documentation/contributing/notifications.md
@@ -36,3 +36,5 @@ The default setting for all contributors is "Often", which means you'll start by
 ## Other notifications
 
 Contributors also receive notifications with status updates about the notes they've written and rated. At this time these are not configurable, but we plan to add more controls in the future.
+
+Authors of posts that are showing a note will also receive a notification about the note, once the note has been consistently showing for 6 hours without appearing or disappearing (indicating its status is relatively stable).
diff --git a/documentation/contributing/top-contributors.md b/documentation/contributing/top-contributors.md
@@ -12,13 +12,12 @@ Top Writers are contributors recognized for writing a significant number of note
 
 Top Note Writers get access to:
 
-**Writing notes about media**
-Top Writers can write notes about media featured on multiple posts, keeping many more people better informed. [Learn more](./notes-on-media.md).
+* **Writing notes on images & videos:** Top Writers can write notes about media featured on multiple posts, keeping many more people better informed. [Learn more](./notes-on-media.md).
 
-**Priority for note alerts**
-Top Writers’ note proposals are more likely to trigger notifications to get rater’s attention.
+* **Priority for note alerts:** Top Writers’ note proposals are more likely to trigger notifications to get rater’s attention.
 
-**Badge in alias profile**
-Top Writers get a badge in their Community Notes profile.
+* **Badge in alias profile:** Top Writers get a badge in their Community Notes profile.
+
+* **See Community Note requests:** Top Writers can see when people on X have requested Community Notes on a post — this can help identify where proposed notes might be found helpful.
 
 A contributor’s Top Writer status can always change as their notes are rated by others.
diff --git a/documentation/images/note-requests.png b/documentation/images/note-requests.png
diff --git a/documentation/under-the-hood/download-data.md b/documentation/under-the-hood/download-data.md
@@ -138,9 +138,9 @@ As we iterate and improve Community Notes, we will occasionally make changes to
 | `noteId`                              | Long   | The unique ID of this note.                                                                                                                                                                                                                                                                                                                                                                          |                                                                                |
 | `participantId`                       | String | A Community Notes-specific user identifier of the user who authored the rating. This is a permanent id, which remains stable even if the user changes their username/handle.                                                                                                                                                                                                                         |                                                                                |
 | `createdAtMillis`                     | Long   | Time the note was created, in milliseconds since epoch (UTC).                                                                                                                                                                                                                                                                                                                                              |                                                                                |
-| `timestampMillisOfFirstNonNMRStatus`  | Long | The timestamp, in milliseconds since epoch (UTC), of when the note got its first status besides “Needs More Ratings”. Empty if the note never left “Needs More Ratings” status.                                                                                                                                                                                                                      | 1 if “Yes” is selected, 0 if “No” is selected                                  |
+| `timestampMillisOfFirstNonNMRStatus`  | Long | The timestamp, in milliseconds since epoch (UTC), of when the note got its first status besides “Needs More Ratings”. Empty if the note never left “Needs More Ratings” status.  | 
 | `firstNonNMRStatus`                   | String | The first status the note received when it got a status besides “Needs More Ratings”. Empty if the note never left “Needs More Ratings” status.                                                                                                                                                                                                                                                | "", "CURRENTLY_RATED_HELPFUL", "CURRENTLY_RATED_NOT_HELPFUL"                   |
-| `timestampMillisOfCurrentStatus`      | Long | The timestamp, in milliseconds since epoch (UTC), of when the note got its current status, including “Needs More Ratings”.                                                                                                                                                                                                                                                                                 | 1 if “Yes” is selected, 0 if “No” is selected                                  |
+| `timestampMillisOfCurrentStatus`      | Long | The timestamp, in milliseconds since epoch (UTC), of when the note got its current status, including “Needs More Ratings”. This is equivalent to: when the note was last rescored.   |
 | `currentStatus`                       | String | The current status of the note.                                                                                                                                                                                                                                                                                                                                                                      | "NEEDS_MORE_RATINGS", "CURRENTLY_RATED_HELPFUL", "CURRENTLY_RATED_NOT_HELPFUL" |
 | `timestampMillisOfLatestNonNMRStatus` | Long | The timestamp, in milliseconds since epoch (UTC), of when the note most recently received a status of either “Currently Rated Helpful” or “Currently Rated Not Helpful”. This value will be the same as timestampMillisOfFirstNonNMRStatus if the note has never switched status after receiving its first non-”Needs More Rating” status. Value is empty if the note never left “Needs More Ratings” status. | "NEEDS_MORE_RATINGS", "CURRENTLY_RATED_HELPFUL", "CURRENTLY_RATED_NOT_HELPFUL" |
 | `latestNonNMRStatus`                  | String | The latest status the note received, when it got a status besides “Needs More Ratings”. Value is empty if the note never left “Needs More Ratings” status.                                                                                                                                                                                                                                              | "", "CURRENTLY_RATED_HELPFUL", "CURRENTLY_RATED_NOT_HELPFUL"                   |
@@ -152,7 +152,14 @@ As we iterate and improve Community Notes, we will occasionally make changes to
 | `currentExpansionStatus` | String | The current status, if any, assigned by the expansion submodel. | "", "NEEDS_MORE_RATINGS", "CURRENTLY_RATED_HELPFUL", "CURRENTLY_RATED_NOT_HELPFUL" |
 | `currentGroupStatus` | String | The current status, if any, assigned by the group submodel. | "", "NEEDS_MORE_RATINGS", "CURRENTLY_RATED_HELPFUL", "CURRENTLY_RATED_NOT_HELPFUL" |
 | `currentDecidedByKey` | String | The submodel whose status was used to determine the note's overall current status. | "CoreModel (v1.1)", "ExpansionModel (v1.1)", "GroupModel01 (v1.1)", "GroupModel02 (v1.1)", ..., "InsufficientExplanation (v1.0)", "ScoringDriftGuard (v1.0)" |
-| `currentModelingGroup` | Int | The ID of the modeling group that this note would be scored by, if eligible to be scored by a group model (determined by the modeling groups of its raters, from the user enrollment file). 0 is a placeholder for no modeling group. | 0-13 |
+| `currentModelingGroup` | Int | The ID of the modeling group that this note would be scored by, if eligible to be scored by a group model (determined by the modeling groups of its raters, from the user enrollment file). 0 is a placeholder for no modeling group. | nonnegative int |
+| `timestampMillisOfMostRecentStatusChange` | Long | The timestamp, in milliseconds since epoch (UTC), of when the note's status was last changed. Value is -1 if the note's status has never changed. |
+| `timestampMillisOfNmrDueToMinStableCrhTime` | Long | The timestamp, in milliseconds since epoch (UTC), of when the note first met the scoring criteria to become CRH, but was set to NMR due to the NmrDueToMinStableCRHTime scoring rule. |
+| `currentMultiGroupStatus` | String | The current status, if any, assigned by the multi-group submodel. | "", "NEEDS_MORE_RATINGS", "CURRENTLY_RATED_HELPFUL", "CURRENTLY_RATED_NOT_HELPFUL" |
+| `currentModelingMultiGroup` | Int | The ID of the multi-modeling group that this note would be scored by, if eligible to be scored by a multi group model (determined by the modeling groups of its raters, from the user enrollment file). 0 is a placeholder for no multi modeling group. | nonnegative int |
+| `timestampMinuteOfFinalScoringOutput` | None | For internal use. Timestamp of scoring run. | None |
+| `timestampMillisOfFirstNmrDueToMinStableCrhTime` | Long | The timestamp, in milliseconds since epoch (UTC), of when the note first met the necessary scoring rules to become CRH, but was set to a final NMR status in order to wait for the minimum amount of stable time before finally CRHing the note. |
+
 
 
 ### Ratings

diff --git a/documentation/under-the-hood/helpful-notes.md b/documentation/under-the-hood/helpful-notes.md
@@ -0,0 +1,25 @@
+---
+title: Follow @HelpfulNotes
+description: How to follow Community Notes that people are finding helpful
+navWeight: 3
+---
+# Follow @HelpfulNotes
+
+Want to keep up-to-date with the Community Notes that people are finding helpful? These notes have always been [public](download-data.md), and you can follow them on various “Helpful Notes” accounts:
+
+- Dutch/Nederlands - [@NuttigeOpm](http://x.com/NuttigeOpm)
+- English - [@HelpfulNotes](http://x.com/HelpfulNotes)
+- French/français - [@NotesUtiles](http://x.com/NotesUtiles)
+- German/Deutsch - [@HilfrchAnmerkg](http://x.com/HilfrchAnmerkg)
+- Italian/italiano - [@NoteUtili](http://x.com/NoteUtili)
+- Japanese/日本語 - [@HelpfulNotesJP](http://x.com/HelpfulNotesJP)
+- Portuguese/português - [@NotasUteis](http://x.com/NotasUteis)
+- Spanish/español - [@NotasUtilesES](http://x.com/NotasUtilesES)
+
+### How it works
+
+These accounts automatically repost Community Notes that meet the following criteria:
+- Have been rated Helpful by contributors
+- Have kept their status of Helpful for 6hr+
+- Have a high helpfulness score (0.45+ intercept score)
+- If the note contains terms that contributors have reported as sometimes overwhelming the Helpful Notes timelines ("spam", "scam", "dropship", "drop ship", "promotion") it will have a lower probability of appearing, so as to avoid overwhelming the timeline. This is experimental to determine if it improves follower satisfaction.
diff --git a/documentation/under-the-hood/note-requests.md b/documentation/under-the-hood/note-requests.md
@@ -0,0 +1,28 @@
+---
+title: Request a Note
+description: How the Request Community Note feature works
+navWeight: 3
+---
+# Request a Community Note
+
+People on X can request a Community Note on a post they believe would benefit from one. If there are enough requests, Community Notes contributors will see an alert, and can choose to propose a note. This gives people on X who are not contributors a way to help, and lets contributors know where notes might be found helpful.
+
+![Button to request a note, and banner showing what a contributor sees when there are requests on a post](../images/note-requests.png)
+
+### Requesting a note
+
+- Tap the ••• menu on a post, then tap **Request Community Note**
+- To be eligible to request a note, accounts must have a **verified phone number**
+- Initially, accounts can make up to 5 requests per day. The limit may increase if requests successfully result in helpful notes, or may decrease if requests are on posts that people don’t agree need a note. This helps prevent spam and keep note writers focused on posts that could use helpful notes.
+- *For Community Notes contributors:* During the note request pilot, we are evaluating whether it is beneficial for Community Notes contributors to have both the ability to write notes and request notes. So, initially, 50% of contributors can both write and request, and 50% can solely write.
+
+### Contributors seeing requests
+
+Contributors who have earned [Top Writer status](../contributing/top-contributors.md) will see that there are requests on a post, if there are enough requests. They can also see a timeline on posts with note requests in their [Community Notes tab](https://x.com/i/communitynotes).
+
+During the note request pilot:
+- Requests will show on a post if the number of requests on the post is greater than or equal to MAX(5, number of views on post / 25000)
+- Requests will show for 24 hours
+- For a post to show up in the Note Requests timeline, the post must be recent (less than 24 hours old)
+
+We expect these criteria to evolve, with the goal that requests are frequently found valuable to contributors, and not noisy. The criteria are initially simple during this pilot phase.
diff --git a/documentation/under-the-hood/ranking-notes.md b/documentation/under-the-hood/ranking-notes.md
@@ -305,7 +305,7 @@ For not-helpful notes:
 
 ### Prescoring
 
-1. Pre-filter the data: to address sparsity issues, only raters with at least 10 ratings and notes with at least 5 ratings are included (although we don’t recursively filter until convergence).
+1. Pre-filter the data: to address sparsity issues, only raters with at least 10 ratings and notes with at least 5 ratings are included (although we don’t recursively filter until convergence). Also, coalesce ratings made by raters with high post-selection-similarity.
 2. For each scorer (Core, Expansion, ExpansionPlus, and multiple Group and Topic scorers):
     - Fit matrix factorization model, then assign intermediate note status labels for notes whose intercept terms (scores) are above or below thresholds.
     - Compute Author and Rater Helpfulness Scores based on the results of the first matrix factorization, then filter out raters with low helpfulness scores from the ratings data as described in [Filtering Ratings Based on Helpfulness Scores](./contributor-scores.md).
@@ -324,6 +324,24 @@ For not-helpful notes:
 
 ## What’s New?
 
+**Sep 17, 2024**
+- Lower threshold for coalescing ratings with high post-selection-similarity.
+
+**Aug 21, 2024**
+
+**Aug 12, 2024**
+- Add a 30min delay for notes that meet the CRH criteria ("NMRDueToStableCRHTime") to ensure they stably meet that criteria across multiple scoring runs before CRHing them
+- Add multi-group models
+
+**July 25, 2024**
+- Only score a subset of notes each time we run final note scoring. This doesn't affect what statuses new notes get, but does cause them to get scored more quickly.
+
+**May 31, 2024**
+- Coalesce ratings on the same note from raters with very high post-selection-similarity.
+
+**May 1, 2024**
+- Modify scoring thresholds for the Expanded Consensus trial to raise the note intercept threshold for Helpful notes, increase tag filtering and require a minimum Core or Expansion intercept for Helpful notes.
+
 **April 11, 2024**
 - Split the scorer into separate prescoring and final scoring phases.
 

diff --git a/documentation/under-the-hood/timeline-tabs.md b/documentation/under-the-hood/timeline-tabs.md
@@ -47,6 +47,7 @@ A variety of criteria are considered when determining whether to include a note,
 - Current status (e.g. "Needs More Ratings")
 - Current helpfulness score (e.g. not a low helpfulness score, highly rated by initial raters, possibly nearing the threshold to earn status of "Helpful")
 - Does not have a large number of ratings (such that more ratings could change the note's status)
+- If the note contains terms that contributors have reported as sometimes overwhelming the Needs Your Help tab ("spam", "scam", "dropship", "drop ship", "promotion") it will have a lower probability of appearing, so as to avoid overwhelming the tab. This is experimental to determine if it improves contributor satisfaction.
 
 In this case of alerts, notifications are sent to a random selection of contributors, excluding the note author and those who have already rated the note. Notifications are also limited by the recipient's notification frequency setting.
 

diff --git a/output/prescoring_meta_output.joblib b/output/prescoring_meta_output.joblib
diff --git a/output/prescoring_note_topic_classifier.joblib b/output/prescoring_note_topic_classifier.joblib
diff --git a/requirements.txt b/requirements.txt
@@ -1,6 +1,6 @@
-numpy==1.26.2
-pandas==2.1.4
+numpy==1.26.4
+pandas==2.2.2
 torch==2.1.2
 scipy==1.11.4
-scikit-learn>=1.3.0
+scikit-learn==1.3.0
 pyarrow