In NJAMC 's thread "My hunt for the Automated Holy Grail" echoeversky raises the subject of a 'Quorum' or 'Voting system' where a signal is only triggered after several different indicators giving similar signals. Big Mike and WolfieWolf both quickly say they've been down that path and to avoid it.
(lots more in thread)
I was actually reading the thread for it's insight into ML and AI, and don't trade automated systems like many of you do here but I'm very familiar with the concept/theory and and am relatively well read on the subject even if I have zero first hand practice.
One of the concepts I've always thought would be interesting to test is exactly this concept of quorum/voting and was surprised you both had issues with it. I understand that the more systems you test, the more you over fitting whether you know it or not.
But lets say I have 10 trading ideas. I break my available data into 12 equal parts. I then test each of my 10 ideas on a separate set of data (sets 1-10). I decide to keep 5 of the 10 ideas and test/optimize a quorum/voting system (not optimizing the indicators themselves) on the 11th data set. Surely if I were to then test this system on the 12th (out of sample) data set, I would have avoided a lot of the pitfalls of over fitting?
Hoping to start an insightful/educational conversation. Any all thoughts / comments appreciated. Throwing out a kevinkdog mention as this seems like it would right up your ally.