Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Speeding up NinjaTrader with a RAM drive (ramdisk)
I keep getting requests for people to share their GomRecorder data. A few threads have been started on the subject but not many are following thru it would seem.
I am collecting data from IQFeed via QCollector. All data contains bid/ask and is tick …
Hi everyone. I really enjoy reading this forum and wanted to share with you my collection of ES tickdata (from 2003 - June 2009). They are data from the CME and therefore should be pretty accurate. To download, please go to post#7 , Bigmike has very kindly …
Still, I wonder if storing all that data in NTD files is a good idea, as it is (essentially) a one way street. I guess it depends on your application. If you are doing a lot of backtesting, then of course, but if charting active month contracts, I guess not. YMMV as they say.
Can you specify certain files to load into ram drive or does it have to be a folder? I want to put SSD in trading box (yes, I tried this previously, long story), but supposedly intense reading and writing will shorten the life, so thinking ram drive but I am limited to 16 gigs memory and my SC data folder is 40 gigs. I don't really want to put historical data in a separate folder and have to move it over every time I want to access it.
Using the new setup with historical data on RAMDisk, I just shut down and restarted Ninjatrader with five workspaces. The restart and reload were much faster than I have ever seen before. This really does seem to work.
Re preceding post, using Link Shell Extension it seems to me that you could create a junction leading to a file just as well as to a folder. I just did this for a png image but the only option given by the right click menu in LSE was to drop a symbolic link. It did not give me a choice of what type of symbolic link.
I don't think that the historical data in the ntd files has anything to do with the SQLServer CE database other than being in the same folder. I don't think it's a good idea to put the SQLServer CE database on the RAMDisk. Ninjatrader has said that in NT7, this database is NOT used to store historical data.
To deal with the problem of large amounts of data stored in the ntd files, it is not necessary to put the top level folders data, tick, minute, day, data and cache into the RAMDisk. You could judiciously choose lower level folders from within those folders, for example just the ones for the front month contracts.
The method that I use to move a folder from the Ninjatrader db folder to the RAMDisk is:
Ninjatrader is shut down.
Verified that I have an NTFS formatted RAMDisk Partition set to start up automatically each time the computer starts. Only needs to e set up once.
Save a copy of the folder to an archive on the hard disk if I don't want to risk losing it.
Use Windows cut and paste to move the folder to the RAMDisk drive.
Using LinkShellExtension, right click on the folder in the RAMDisk to designate it as the source of the link and then right click on the db folder to create the junction, which will have the same name as the folder in the RAMDisk partition.
Just did a shutdown and restart with seven workspaces. Most of those workspaces have charts with multiple indicators that use one tick secondary dataseries, and Gom data. Restart only took a couple of minutes.
This is the best startup performance I have ever seen, by far. Before I set up the RAMdisk a seven workspace instance would usually not restart at all.
Will need to wait until Wednesday to see if improvement continues when markets are active.
Thanks @pawnbroker, this is one of the best Ninjatrader performance improvements ever.
I tried putting the Templates folder and the SQLServer CE database on the RAMDisk but did not see any performance improvement, so moved them back to their default locations. Will try that with the Indicator folder too, mainly out of curiosity.
For now the RAMDisk contains only the five data storage folders (data, day, tick, minute, cache) from the db folder, and the db folder has junctions that link to them. I used Link Shell Extension to create the links.
Technically, the database files are inside the SQLServer CE folders called Ninjatrader and Ninjatrader.Old. The files in the five folders are not part of the SQLServer database. They are data storage files, but strictly speaking are not database files. When Ninjatrader refers to the database they are referring to the SQLServer database, which does NOT store historical data.
This seems like a stable setup. The RAMDisk software is a production version so I don't consider this to be a white knuckle "experiment". Ninjatrader loads faster and with less drama, especially with multiple workspaces. There is none of the flashing and freezing that I was used to seeing as workspaces and charts loaded from data stored on the hard disk. The RAMDisk backup image is only being updated once per hour and on shutdown, but the loss of some of that data due to an unexpected event would not be a serious problem anyway.
The main problem is that the datastore folders will eventually outgrow the RAMDisk partition. There are various ways to handle this, which may be a little awkward but are worth the trouble based on the improvement in performance.
Putting the datastore folders on their own dedicated physical disk, preferably an SSD and linking to them from junctions in the db folder would simplify dealing with large amounts of historical data. This would not perform as well as the RAMDisk but would certainly be an improvement on the base configuration due to reduction of I/O conflicts - even if using a hard drive.
You may like to try SuperCache 5 to create a RAM cache for a drive. I tried that while the markets were closed and saw no improvement, but I saw the same with a RAM drive too. Perhaps my set up needs the stress of live data coming in to get an improvement.
SuperCache sounds good, because you can dedicate say 2 GB of memory for a disk of any size. That would avoid the disk space issue, but it is only worthwhile if there is a performance gain.
I have not tried FancyCache yet, which is a similar product in beta and free while in beta.
With both of those apps, you should not use the option to cache writes. That is very dangerous as lost writes can cause corruption and NT will be improved most by fast reads.
Anyway, I am glad to see that the RAM disk idea is having some benefits!
I'd been getting increasingly frustrated with NT's slow startup. After reading this thread, and particularly Zondor's posts, I decided to try reducing the size of the db folder by archiving what I wasn't using. By itself, this cut startup time by about two thirds. Perhaps NT runs a check on the entire db on initial startup.
I must try that one. I've been deleting the database cache which seemed to help.
This has been a bone of contention over the years. I'm hoping NT8 overhauls the entire database structure. I suspect Ninjatrader doesn't have a hardcore database expert on staff who has the skills to redesign the architecture. It does seem extremely slow particularly compared to other packages I see people using in demos. I'm seeing FT71 loading his volume profiles back a few years in seconds....
Have not given much thought to disk caching software since I am already dedicating two USB memory sticks to Windows Ready Boost®.
Noticing that with the historical data archives moved to the RAM disk I can run multiple complex workspaces simultaneously in replay mode, reliably. Up until now this would cause Ninja to freeze and crash. However with this setup, during day session hours the CPU load is very high and with the contoller set for 500x, the replay is only at around 3x actual speed.
If using inefficiently coded indicators the results would not be as good. Maybe we should call the revision of indicators to reduce resource load "economization" rather than optimizaion, to avoid confusion with strategy optimization.
