Meme of the Month
by Keno
Diagnostic Tool Update and Numerai-Tools 0.0.13 Released
Updates to the Diagnostics tool and the Numerai-Tools python package have been made to reflect the recent scoring changes.
See below for a summary of the changes to Numerai-Tools from Ark:
Rename contributive_correlation to correlation_contribution for clarity and update some docstrings
Add logic to correlation_contribution to re-scale targets to match internal data
Add top_bottom functionality to functions used by diagnostics
Add max_feature_correlation function
For the Diagnostic Tool, the main change is that it will now include Benchmark Model Contribution (BMC).
Some reasons for the change are outlined in Ark’s Discord post:
The Numerai Diagnostics Tool has been updated to now include BMC. This BMC is calculated against the predictions of a single benchmark model: “v42_teager_plus_cyrus”. We chose to include BMC against this specific model because it could be a useful metric when trying to optimize for MMC. Enjoy.
Why BMC?
Good correlation contribution against our favorite benchmark model is more indicative of an edge over our models than MMC.
What about MMC?
There is a natural bias towards older data sets in the historical Meta Model because that’s what was available at the time. Showing MMC against the historical Meta Model won’t tell you how good the model is when everyone is on modern data with modern models. Because of this we won't be including MMC in diagnostics.
Check out the Target Ensemble tutorial for example usage of Numerai-Tools!
Model Slots
The maximum number of models you are allowed to create in your account will now be tied to your Season ranking, moving away from a fixed allocation to every user.
More details from Ark’s post in Discord:
The model limits will be determined by your highest earned tier:
novice (default): 30
apprentice: 60
contributor: 90
researcher: 120
expert: 150
master: 200
grandmaster: 250
Will this delete any of my models?
No. Any account with more model slots than their tier allows will have their slots maintained.
Why these limits?
We looked at how many models each account has and set the limits above the 90th percentile number of models created by active accounts in each tier. This means the vast majority of data scientists in that tier should have more than enough slots to experiment and learn.
With the transition to MMC, which is locally computable, we want to encourage data scientists to optimize models locally (or with the diagnostics tool) rather than using live models.
We want to give new users a sense of long-term progression so unlocking things like model slots with your GM tier helps to achieve this.
Numerati SF Holiday meetup
User Unsentient hosted a Numerati SF meetup over the holidays with a great turnout of over 20 people!
Hopefully some of them added Numerai Grandmaster to their New Year’s resolutions!
NumerGPT
CoE member surajp has created a Numerai community bot with knowledge of the docs and forum posts.
The model was made by fine tuning the EleutherAI/gpt-neox-20b baseline model with data scraped from the forum, the official documentation and Rocket Chat archives.
As seen in the demo below, NumerGPT can answer questions about Numerai specific topics such as Numeraire!
If you have a ChatGPT Plus subscription, you can try it out here!
More details about NumerGPT, including the notebooks used to create it, can be found in Suraj’s forum post.
CoE Transaction Report
Dec 6th 2023 -55 NMR Numerbay
Dec 10th 2023 -38.7 NMR CoE Newsletter
All transactions from the CoE wallet can be viewed at the following link using SAFE.
Disclaimer: This is not an official Numerai newsletter. It is sponsored by the Numerai CoE, a decentralized autonomous organization. Every effort is made to provide accurate and complete information but there are no claims, promises or guarantees about the accuracy of the contents.