The CoE Numerai Newsletter- Edition #16
Nov 1st "Fireside Edition" Q4 2022
by restrading
Daily Tournaments Are Here
On Oct 21st 2022 it was announced by the Numerai team that Daily rounds for the Numerai main tournament and the Numerai Signals tournament would begin the week of the 24th. A forum post was also dropped by the team.
TLDR
daily rounds on Tournament and Signals starting next week
no code changes required to participate
payouts coming soon
Why daily tournaments?
Numerai currently runs on a weekly cadence – weekly tournament, weekly meta model, weekly trading.Research has shown that daily trading will improve the hedge fund’s performance and capacity. To support daily trading, we need a daily meta model, which in turn requires a daily tournament.
Getting to daily trading will be a huge milestone for Numerai and we need YOUR HELP to get us there!
What is changing?
Starting next week, Numerai Tournament and Numerai Signals will open 4 new rounds a week on Tuesday, Wednesday, Thursday, and Friday, in addition to the current rounds opening on Saturdays.Specifically, this means that after round 339 opening on Saturday October 21st, round 340 will be opening on Tuesday October 25th, round 341 will be opening on Wednesday October 26th, and so on.
The open time and submission deadline of the rounds opening on Saturdays will stay exactly the same as before. The new rounds opening on Tuesday-Friday will open earlier at ~13:00UTC and have a shorter 1 hour submission window.
What do I need to do?
If you don’t want to participate in these new rounds, then there is nothing that you need to do. You can continue to participate once a week just like before.If you do want to participate in the new rounds, then you can use the same* data and submissions API to run your daily model pipeline. And if you are already automated with Numerai Compute, then you can simply enable daily triggers on the models page of the website.
*V2, V3, V4 data files and signals universe will be available daily via the API but the legacy dataset zips will only be available for Saturday rounds and not the new weekday rounds. Validation data will continue to be updated weekly and not daily.
Getting ready for the 1 hour submission window
If you are not yet automated, we strongly recommend that set up Numerai Compute. We are also happy to help with integrating any other set up you have (notebook, cron etc).If your model just takes longer than 1 hour to run but you still want to participate, don’t worry. We will soon support a feature to let your late submissions automatically count towards the next round.
Payouts coming soon
Initially, there will be no payouts for these new rounds and so the payout factor threshold for these rounds will be set to 0. This is meant to give everyone time to adjust their model pipelines and for us to fix any issues based on your feedback.Once the system stabilizes, we will gradually decrease the payout factor threshold for rounds opening on Saturdays and increase the payout factor threshold for these new rounds opening Tuesday-Friday, keeping the overall payout factor the same, until we reach about a 70/30 split.
We expect this change to increase overall payouts (and burns!) due to a faster rate of compounding. We also expect the payout factor for these new rounds to be much higher for a period of time due to lower number of models participating.
More details regarding the upcoming changes to the payout factor threshold will be separately announced.
On Oct 25th daily submissions went live. It was a smashing success with a final submission count of 1938 for Tournament and 319 for Signals!
We were also fortunate enough on the 28th to be joined on our weekly CoE twitter space by Ark from the Numerai team to go over mechanics live and answer all community questions. You can still listen to the recording and it will be available until Nov 4th.
There was a lot of Rocket Chat volume. I sorted through to find a couple of the longer threads with the most info. See below
https://rocketchat.numer.ai/channel/general?msg=6dCq3pau3JTLF2equ
https://rocketchat.numer.ai/channel/general?msg=Frw7nXkdgTauxyEA9
NMR Monthly Performance
We saw some volume spikes towards the beginning of the month moving the price from around $14.50 to the $18-20 range but did not hold up. Overall NMR was fairly flat ending around $15.
Updates To The Signals Targets File
- Column rename: `target_factor_feature_neutral_20d` to `target_factor_feat_neutral_20d`
- 20D targets on 2022-09-02 now have non-NaN values
[Updates] NumerBay - The Community Marketplace
2022-10-09 — 2022-10-23:
Added temporary workaround for round numbering to keep NumerBay weekly (See list of closed issues)
To add support for daily tournaments soon
NumerBay will remain weekly for now:
Orders that cover until round 339 will remain unimpacted
Quantity for new orders still refer to the number of weeks
Past and new orders that extend beyond round 339 will have their round span in increments of 5 from 339 onwards. E.g. a purchase of 3 rounds at round 338 will be valid until round 344 (338, 339, 344)
Webhooks are still triggered weekly on weekends only
Missing submissions for the daily rounds in-between don’t count towards the on-time reputation (not implemented yet but will do so soon)
CoE Transaction Report
Oct 18th 2022 -27.5 NMR Newsletter
Oct 23rd 2022 -31 NMR Numerbay
Oct 25th 2022 Claim SAFE token
Numerai Q4 2022 Fireside Chat
1:09
Richard: I'm doing well. I'm back in San Francisco. I was in New York for about six weeks working on fundraising for the company and our hedge funds so we are AUM is now 120 million dollars so we have made quite a bit of progress there I think it was seven million dollars 18 months ago or something so we're definitely growing the growing the fund. We also just hit our three-year track record about one month ago and when you have a three-year track record and more than 100 million in AUM it's sort the right time to go marketing the fund. In addition, almost every asset class in the world is down very badly this year whereas Numerai funds are not so that's been a big focus for me this quarter how.
2:34
Anson: I think me sort of similar story I think since the last time we did this fire side chat I spent a lot of time helping to just productionize and scale up the hedge fund infrastructure so as Richard said you know we had a lot of AUM growth I think with that comes a lot of operational and system complexity so I've been working closely with our hedge fund team to kind of just scale that out and make sure that we're ready that we were ready for the money coming in and we'll be ready for the next phase of money coming into. I've also worked a little bit on hiring so we have a new front-end engineer Daniel who joined us two weeks ago really excited to kind of switch back switch gears here and like spend a lot of time working on the tournament this quarter so everything from the website to doing stuff with like signals with Richard. The web site we have a new homepage on numer.ai and then we're even working on a new one for signals. A lot of exciting projects coming on the website side so excited to share that with you guys too.
3:59
Anonymous, how does numeri verify that TC is achieving the goal it is supposed to for both the user and the fund perspectives? I guess what is the goal of TC? TC is true contribution. it is our latest and greatest way of scoring users so the way I explained it to I guess people outside of Numerai is core or you know fnc or these other like standard scores are a way for us to measure your personal performance how is your model doing against the target that's an absolute thing and it only measures how well your own individual model is doing. TC is sort of like measuring what is your marginal contribution to the group's performance measured at like an end-to-end level on like the impact your model's predictions had on the funds returns so not only does it take into account like how your model ensembles and interacts with other people's models it also takes into account all the downstream components in in Downstream pipeline between your model's predictions and actual p&l of the hedge fund so exactly that's what TC is so how do we verify that TC is achieving the goal it's supposed to. One of the things it's trying to do is pay people who are contributing the most to the hedge fund performance which it's doing a perfect job of by definition. Some things we see that are very encouraging is someone's staking a lot on a model with 30% correlation with the meta model and having incredible TC. There are some very very very good models that are very unusual and uncorrelated so that does appear to be working that the TC is encouraging people to make more unusual models. I think another thing that I don't think
7:57
We have actually been doing some thinking about TC and research on it so we made three new versions of testing one is called STC which is supreme TC. We have two funds Numerai one and supreme and they both have different Optimizer settings so they both have different true contributions and so we wanted to make a TC version of that and then we wanted to make a 60-day version of TC for both of them so that's why we have four TCS now that actually have been backfilled in our database and we were thinking maybe some new version of TC should be all of those.
9:03
As it turns out if you take the all four of these TCS added together the volatility of the different TC's Nets out and the average person has a much higher Sharp on their TC. Some people might get worse though on this measure so we are going to keep doing the simulations we were just discussing. We haven't done that full simulation with this new kind of Ensemble TC but we believe it will like very likely be good and we are doing it right now. I think it's still in research and development how do we make TC better and have the average user have a higher sharp without making it not true. I think maybe one more thing worth mentioning is on the user facing side we're always thinking about how do we help users get more TC so any type of scoring metric or maybe even new targets that we research we asked ourselves the question will this help users get TC.
13:40
Joaquin asks any update or timeline for the new Numercon announced features like Benchmark models, synthetic data, stake weighted meta model on the leaderboard and staking 3.0? So, we did announce a lot of features and the biggest thing we announced at Numercon from the business perspective was Numerai Supreme. It’s quite hard to raise money for this fund because it's quite a sophisticated product. It doesn't appeal to everybody and it's not suitable for everybody but Supreme actually started with more than 50 million dollars so we did a very good job on the Supreme announcement. Getting started with that new fund product but then the other things kind of took a little bit of a back seat because it's quite complicated to do two funnels so the one thing listed here is Benchmark models. Benchmark models is kind of an interesting and useful way to know if you're going to get TC so for example we have many Targets, one of them is Jerome 60. If you could see a Numerai created model that's created on Jerome 60 and backfilled over the last three years and you could see it's TC every week for the last three years then that might encourage you that might encourage you to use that Target but you can't see that but we have been working on that. The first part that involved was us was building our own internal model system that was actually a proper piece of infrastructure. We just haven't yet put it into the websites but I hope to do that soon and I don't think I'm not sure it'll be this quarter but we will definitely be working on it and I think it will be very helpful to people. Similar story with the state-weighted metamodel on the leaderboard. I think we know it's something that you guys want a lot it's something that I think would really help the community you know kind of understand what's going on and also just you know maybe share in the joy of watching State weighted meta model do really well but yes so we want to give out the state-weighted model predictions. Working on that as well. Speaking 3.0 maybe I can give an update here. When I announced it I had basically had a proof of concept built out that were that essentially hard-coded the rules as we had the staking and payout rules as we understood it back in April but obviously since then a lot of things have changed. We started looking into longer time Horizon targets like the whole 60 Day stuff which fundamentally changes the game of you know the whole delay system of staking and unstaking so that kind of got us going back to the drawing board. I think like as we're thinking about these things like TC a little bit more and all these things that would potentially change the payout system it just seemed to me like it's not it wasn't the right time to launch 3.0 also you know we were busy building the new hedge fund but I still think that like there are a few big changes coming up that will fundamentally change things. We should not put the staking payout rules on chain just yet so I guess my answer is it's coming soon. For synthetic data we had this pretty long thread of people talking about synthetic data and actually Suraj recently sent me something kind of interesting using stable diffusion to make synthetic data from panel data but I don't know I mean it's also something you guys can do so we didn't want a person we have an allocated time to it for this coming quarter but we are still watching all that research and watching that Forum post and thinking about how we can do it at some point.
21:10
Restrading, will there be documentations for how the additional targets were constructed? Yeah I feel like the probably there isn't enough about the other targets and there's probably more we can share. Often they're just neutralized to different things I don't think we're going to get quite the way of saying this one's neutralized to momentum and this one's not or I don't mean to go that far but we can definitely say how you know we would use them and have how you can think about them so potentially when we have even more targets we can do that better. There is a new Target we're working on which is better than Nomi which we haven't had been able to make a Target better than Nomi very easily. We had some ideas that we've been trying and I think it's going to be very useful one for people to look at and possibly will replace Nomi. We still don't know timeline yet but we're soon gonna finish that Target and get it out to you.
23:47
Anonymous asks when daily tournaments?
SEE ABOVE
26:18
Steve asks when more model slots? Haven't thought about this for a while I mean currently it's at 50. We can probably do more. I think right now the reason why I can't just do 100 like right now is because the models page is not designed that well and if you actually have 100 models the page will probably freeze if you try to like load it and stuff so there's a little bit of stuff we want to do to upgrade the website. Very soon.
27:34
Correlator asks can all the metrics like tb200 different Target cores for 20-day 60-day be made available via the API so we can study TC ourselves rather than relying on the team? I think we should yeah I mean if we're making it's not that computationally hard. We also had to sort of massively upgrade our scoring architecture in order for us to be able to nicely compute scores in different time Horizons and do them all in parallel because they're going to take way too long so yeah I think now that we have the system we've built it it's in production we can easily sort of just give you everything that you need. Coming soon.
31:59
Anonymous if Numerai adds signals submissions to The Meta model how much would that change the TC of the main tournament? Isn't this important to see? I guess the question assumes that we're Computing TC across both signals and the regular tournament together. So right now, the way TC works for signals is it is comparing you versus The Numerai Meta model and actually ignoring the other signals submissions. It's just the way we wanted to start it because we wanted it to be like Numerai’s TC and Numerai’s TC is not including the signals models at the moment so that's kind how we've done it so far but yes at some point I think there will just be something called The Meta model which is everything and it might be just State weighting all the submissions from signals and main so once we have something like that then maybe all the TC’s will be based on that system. I quite like the idea of TC just being kind of against Numerai’s metamodel not including signals.
34:14
Restrading, if we move to daily tournament with a three-month resolution will stake decrease take three months to release or release immediately? So this is sort of the the proposal that we almost started which was if you had a three month or I guess a 60-day resolution it would extend your stake decrease delay period to three months or to that 60 days. We're kind of doing final reviews we were just thinking we didn't really like that system in terms of its ux so we're not doing that anymore. What we want is something much closer to immediate releases I think and that's sort of like the promise of staking 3.0 or at least partially which is allowing you to manage your staked NMR sort of like move it around in a faster way. Our current idea is that if we can have scores that actually compound nicely every single day then we would be able to have faster releases so that's something that we're researching right now and that we're designing towards.
37:20
Boar, can we get another what correlates with TC analysis from Michael Oliver? I'm assuming that as we optimize for TC what correlates with TC also changes. Yeah I mean this kind of I want to say, right you don't know what our features are but there is a way to test historic data how well you would do so in Diagnostics, Diagnostics pre-neutralizes you and gives you your result and so the fact that you can actually do this potentially you can compute this type of thing yourself like see what correlates with TC more easily but yeah we're still developing TC as well and so I don't want to make like a lot of details on how to get good at current TC.
39:05
Anonymous asks when will we see Numerai Supreme's performance? When it's kind of been long enough. It doesn't make sense to show like a sharp or something over a two-month period so I think just when it's when it's long enough
43:26
Anonymous is signals still a focus for Numerai? Absolutely! It's becoming a very important thing and it has over a thousand stake submissions so it's one-fifth the size of Numerai. It's critical. There's no world where we don't have signals because we need a way for people who have their own data to create signals and send them to us
48:42
On a longer-term Horizon what kind of potential drastic changes to the classic torment are you exploring? Well daily tournament's coming soon. New data also coming soon. New targets coming soon.
54:33
For daily predictions will you need multiple predictions per 24 hours depending on Market time zone? No we're doing one prediction one submission window per day for now.
54:51
Do you have an estimated expected monthly cost for daily compute light? Okay so it's pennies probably dollars if we're doing it every day seven dollars something like that on the order of like under 10 bucks but it depends on like what resources you're actually doing. Okay let me take a step back. Compute light is a new sort of like version of compute that we're building which is much much easier to deploy than our old version which of compute which we call it compute heavy. It is much more restrictive though in terms of like what you can actually deploy. Compute heavy gives you full flexibility. Compute light sort of gives you very tight boundaries as like what you can and cannot do but in return it makes the deployment process much much easier so I think I imagine the onboarding process for Automation in the future to be like if you want something quick and easy then you can use this compute light. If you want like fully featured maximum flexibility then you can use this like the compute heavy framework. Monthly cost for both of these things it heavily depends on how much resources your model or your scripts when invoking these models is. I think my approach is to just be able to like help you do cost optimization to help and cost tuning by giving you good tools and good metrics on the website itself but okay all that being said compute light probably still be on the order of dollars per month.
57:27
Restrading does TC and fnc v3 calculations currently involve the dangerous features? I think they might actually. I hadn't thought of this. That's a good question. The features were very good in the past and then they became noise that's how to think about it and actually if you neutralize by noise is always going to be orthogonal to you already. It was it's really about training your models you don't want to train on these dangerous features.
1:00:41
Anonymous asks did the data augmentation form post bring any value? So that's the synthetic data question okay we still think it's going to happen in some way. We're gonna probably gonna do it eventually but we also think other people can do it right now.
1:01:00
Why did you roll back the universe expansion? Ah you noticed yeah, we tried to make our universe bigger and add some more stocks and then there was actually a bug in the research. We changed it. So usually adding more to your universe is going to help you but what we cared about is that expanding the universe would help even if you didn't retrain your model on a bigger universe and that wasn't really true so we decided to roll it back until some point where maybe we give you a much bigger Universe to train on.
1:02:02
When will Numerai be profitable? Where is the break-even point? How solvent are you? Great question we actually are profitable. By the way accounting works whenever we've sold NMR that creates Revenue so if we sell NMR to a crypto fund like Pantera or Polychain or Paradigm then we get Revenue. It’ss not like selling shares and so we have this Revenue in so actually we've been profitable for many years on that but very not good basis so we want to get to real profitability where the fee income from our hedge funds pays for everything. It depends on our performance but we'll probably make a few million dollars this year in Revenue. Maybe it could be four million dollars but we have about seven million in costs but I think next year will be profitable for sure not for sure but probably for sure.
1:03:27
Correlator can changes to the scoring metric be made at regular intervals say twice a year in January and July with minor updates in between? If required this would reduce uncertainty. Yes I think one of my favorite ideas which I learned from watching this this one video on how to design computer games but essentially is to have more predictable road maps um predictable schedules for data releases and changes to the scoring and the payout function and system. I think we'll get there eventually although I think right now we're sort of in the phase where you know we have these ideas on how we can improve right and I think we want to hold ourselves to like if we find something that is much better for everyone we want to release it like as soon as possible. I think the solution to that is like a scheduled like regular interval update system. I think we'll get there eventually. I don't know if we're going to do it like right now but maybe I think eventually we have to do that yeah.
1:06:50
QE any chance we will see core x2 in the main tournament just like with signals? No
1:07:16
TC is murky, can you consider moving TC in-house and using it to create concrete scoring metrics for us? Oh like we make special metrics yeah I think that's something we thought about right it's like we would compute TC internally then we would change the scoring function to like the best approximation of TC using scores that people can actually optimize for but I don't think that's the right thing to do. Yeah the right thing to do is to score on TC even if it's black box.
1:08:42
Restrading how was the medium feature set selected? It’s just a good subset that sort of already has really quite a lot of information in the data so I would say very very good models can be built on the medium feature set like ignoring the other features. Actually there's an incredible presentation at the New York event by Ai_Joe and he basically has a way of filtering out features by looking at the sharp of each feature individually and found that when he drops those features he can make much better models.
1:10:42
Anonymous when 24 7 support? We already have 24 7 support or we try to. Maybe you can give me some feedback on what more do you want to see
1:11:00
Anonymous why not continuous payout multipliers? Do you use these multipliers somewhere in the med model construction? We don't currently use the multipliers in metal model construction but we may in the future. Something we could look into continuous payout multipliers mainly a ux issue I just didn't want people to be sort of like playing that meta game of like adjusting it by .01 like every week based on that their scores or something. Although the back-end system that we built definitely supports continuous payroll multipliers. Maybe one day we'll get there.
1:11:48
Correlator have you thought about alternative data which might be industry specific and available since past few years? Amazing question Yes so yeah we basically kind of had this rule like we're not going to add data unless it goes all the way back to the beginning of the data which is like 20 years and it better apply to all the stocks in our universe. You know machine learning requires a lot of data and so if you're going to start cutting off you can have data that starts in 2020 and it's not going to be long enough to be useful. We will be releasing new data this quarter and new features. We are very excited about new data sets that might break those you know those same rules that we self-impose.
1:13:58
Anonymous since TC relates to portfolio construction develop another data set that has covariance data so that we can create model portfolio construction as our second step? I like the question. A big thing in Quant is portfolio construction and Numerai isn't about that, it's about making signals but I do think in fact there's something I kind of want to do quite soon which is on signals you could you just upload a weight of a portfolio that you like based on your signals. I don't really think it's very easy to give covariance data for numeri but I think for signals it's probably a good place to let people play with portfolio Construction. Maybe in a few years from now we can find a way to use it.
1:15:43
Will signals submission be daily as well? Yes
1:15:48
Anonymous is Numerai hiring? Are there any positions open to contribute towards daily predictions and the long-term decentralization of everything? We're always hiring and we're always looking for the best people in all teams and all positions so if you want a job here you think that you have something to offer and contribute just email me slyfox.
1:16:59
QE when staking on core J60? I don't think do that. Jerome 60 is just a Target to train on and we think it's kind of a nice one it's sort of neutral in a different way than the other ones so it's worth kind of like checking if it helps to train models on it if you haven't already.
1:18:23
QE any chance staking 3.0 will include rebalancing stake amounts between staked models when 3.0 is released in the next couple of months wink? Yeah I mean I think we talked about that again so like yeah I don't like rebalancing now I think the true answer is just letting you do releases faster so you could do whatever you want. You can rebalance or you can just take it out.
1:18:53
API internal server error no is there a chance we'll get decent error messages in the future? Yes if you're getting API internal server errors please reach out to me or Chris and we will fix those as soon as possible.
1:19:05
Correlator if Numerai could incentivize using AWS credits or providing internal compute resources like GPU that would be cool. How about compute credits for metal winners? I like that actually. We actually spoke to AWS a couple of years ago when we initially did compute and we almost did this entire system where we would give out compute credits as sort of like as part of the payout system but I think we were just not big enough at that point to Warrant them sort of investing a lot of resources into helping us achieve that. I think we are big enough now or maybe we need to get a little bit bigger before we can get like group discounts for AWS credits but I think it's a good idea
1:20:16
Wigglemuse news regarding daily submissions and deadlines? Maybe we can get a semi-persistent row IDs changed weekly so we can use previous days data for time consuming predictions I can see? Yeah maybe. I haven't thought about it much. Yeah, I haven't thought about this. I think I'm really interested in hearing more from you guys about what kind of time consuming pipelines you have on your end so that I can help you around these issues. I'll have to talk to Mike P about the row ID thing. I don't know if I can promise you actually getting the semi-persistent one.
1:21:53
Anonymous V4 data has an extra 141 features 1191 in total compared to in V3 we have 1050. Can you give us some idea on how the new 141 defer from the 1050? Yeah I don't actually know enough details actually myself about that. As a general rule though we're never going to tell you what the data is on Numerai.
EMAIL LIMIT HIT. PLEASE SEE VIDEO FOR THE REST OF THE FIRESIDE. SORRY :(
Disclaimer: This is not an official Numerai newsletter. It is sponsored by the Numerai CoE, a decentralized autonomous organization. Every effort is made to provide accurate and complete information but there is no claims, promises or guarantees about the accuracy of the contents.