NexusFi: Find Your Edge


Home Menu

 





Tick Database Storage


Discussion in Platforms and Indicators

Updated
      Top Posters
    1. looks_one Big Mike with 10 posts (1 thanks)
    2. looks_two MXASJ with 7 posts (0 thanks)
    3. looks_3 phyzfer with 5 posts (0 thanks)
    4. looks_4 darthtrader3.6 with 2 posts (0 thanks)
      Best Posters
    1. looks_one gomi with 1 thanks per post
    2. looks_two samurai with 1 thanks per post
    3. looks_3 Adamus with 0.5 thanks per post
    4. looks_4 Big Mike with 0.1 thanks per post
    1. trending_up 27,550 views
    2. thumb_up 6 thanks given
    3. group 18 followers
    1. forum 40 posts
    2. attach_file 3 attachments




 
Search this Thread

Tick Database Storage

  #11 (permalink)
 MXASJ 
Asia
 
Experience: Beginner
Platform: NinjaTrader, TOS
Posts: 796 since Jun 2009
Thanks Given: 109
Thanks Received: 800

This outputs to the Output Window OK, going back the number of days specified in "Days to load" when you start the strategy. Next hurdle is getting to write a CSV file to disk:

 
Code
                            
#region Using declarations
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Data;
using NinjaTrader.Indicator;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Strategy;
using System.IO;
#endregion
// This namespace holds all strategies and is required. Do not change it.
namespace NinjaTrader.Strategy
{
/// <summary>
/// Enter the description of your strategy here
/// </summary>
[Description("Strategy to write historical data from BarSeries[0]")]
public class 
DataWriter Strategy
{
#region Variables
 
#endregion
/// <summary>
/// This method is used to configure the strategy and is called once before any strategy method is called.
/// </summary>
protected override void Initialize()
{
CalculateOnBarClose true;
BarsRequired 20;
 
}
/// <summary>
/// Called on each bar update event (incoming tick)
/// </summary>
protected override void OnBarUpdate()
{
PrintWithTimeStamp(Open[0] + "," High[0] + "," Low[0] + ","Close[0]);
}
#region Properties
 
#endregion
}

Having a bit of agro with StreamWriter. This one only writes one line, though the Output Window looks OK. Here is the code;

 
Code
                            
{
//Print the data to the Output Window 
PrintWithTimeStamp(Open[0] + "," High[0] + "," Low[0] + "," Close[0] + "," Volume[0]);
 
//Write the data to PathFileName
 
using (StreamWriter sw File.CreateText(PathFileName))
{
sw.WriteLine(Time[0] + "," Open[0] + "," High[0] + "," Low[0] + "," Close[0] + "," Volume[0]);
sw.Close();
}

I've attached the strat as it stands now. NT7.

Attached Files
Elite Membership required to download: DataWriter.zip
Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
Better Renko Gaps
The Elite Circle
ZombieSqueeze
Platforms and Indicators
NT7 Indicator Script Troubleshooting - Camarilla Pivots
NinjaTrader
Cheap historycal L1 data for stocks
Stocks and ETFs
What broker to use for trading palladium futures
Commodities
 
  #12 (permalink)
 MXASJ 
Asia
 
Experience: Beginner
Platform: NinjaTrader, TOS
Posts: 796 since Jun 2009
Thanks Given: 109
Thanks Received: 800

Too late to edit the bad code above, so here is the good code. Its based on something that was already available on the Ninja support site (Duh!). My next step is to offer the option of changing the DateTime format.

This should make my workflow a bit simpler for now at least.

Indicator ExportChartData and sample output attached (NT7).

EDIT: I've completed the indicator (mostly) and posted it in the downloads section under Ninja Trader/Misc. Hope it helps.

Attached Files
Elite Membership required to download: ES_03_10_5min.txt
Started this thread Reply With Quote
  #13 (permalink)
ptd26
Portland, Oregon
 
Posts: 27 since Jan 2010
Thanks Given: 3
Thanks Received: 2



Big Mike View Post
Right, it uses OnMarketData and etc, it does not convert the existing database.

I used to run it on a separate box, a Quad Core Q6600, it would come close to maxing out NT 6.5 single-core/thread, it would be much better to do it on NT7. The SQL db server should be local if possible as well to reduce latency. I was recording for a half dozen instruments.

Mike

Big Mike, what's the granularity of the ticks you are storing? Are you storing OHLC or the individual transactions, like this:

7:24:47.664433 [TRADE] ESH0.CME 1 @ 1096.75
7:24:47.664433 [VOLUME] ESH0.CME 115924 @ 0.0
7:24:47.664623 [BID] ESH0.CME 176 @ 1096.75
7:24:48.065112 [TRADE] ESH0.CME 1 @ 1096.75
7:24:48.065112 [VOLUME] ESH0.CME 115925 @ 0.0
7:24:48.06566 [BID] ESH0.CME 175 @ 1096.75

I spent about 5 minutes looking for a way to do this in NinjaTrader, but was not able to figure it out. I am currently logging to text files, but I plan to store binary files by day. Maybe eventually I will move everything to something like Berkeley DB. Whatever supports the most data compression and reliability.

Reply With Quote
  #14 (permalink)
 phyzfer 
Chicago
 
Experience: Intermediate
Platform: NinjaTrader
Broker: Mirus/Zen-FIre
Trading: GC, ES
Posts: 73 since Mar 2010
Thanks Given: 40
Thanks Received: 33

If I may add my $.02. Saving tickdata is an immensely expensive (data storage, latency and resource HOG) endeavor. Saving to the database, even with a buffer, would make any relational database grow exponentially fast. Further, querying that data with SQL would take forever and is time-prohibitive. I worked on a project where we listened to tick data and saved them to binary-files (similar to how NT saves tick data in NT7) using a buffer (like BM suggested). We then created binary read applications that would be able to translate and chart the data as needed (timebound bars, tick bars, etc).

I would recommend simply opening up a bunch of charts and have NT do all of the heavy lifting with regards to listening and saving the data to binary (have numerous NT charts open but minimized maybe?) and have it store the data in its binary format. Then create some sort of binary-reader that will be able to translate the saved binary files for later use (either in matlab, r or anything else).

Does that make sense? I guess my question is, if you simply open up a 1tick chart on NT, will the data be saved? I don't have a broker-connected NT at the moment, so I'm unable to test this.

Reply With Quote
  #15 (permalink)
 MXASJ 
Asia
 
Experience: Beginner
Platform: NinjaTrader, TOS
Posts: 796 since Jun 2009
Thanks Given: 109
Thanks Received: 800

There is a 4 Gig limit on the size of the SQL CE database engine that NT7 uses, so that would need to be taken into consideration ie you couldn't record tick data of the whole CME Group product list for very long on a single instance of NT ..

A side project I haven't looked into yet is sync between SQL CE and SQL, but I'm looking at that more for getting an overview of multiple NT users positions than for storing tick data.

I'm leaning towards the "buy" camp in the "buy vs. build" for historical tick data if 1 second timestamps are OK.

phyzfer if you need live futures data for development (not trading) try this:

Zen-Fire. The Ultimate Trading Solution.

That will get you login credentials for 30 days or so which you can use with your existing NT installation.

Started this thread Reply With Quote
  #16 (permalink)
 phyzfer 
Chicago
 
Experience: Intermediate
Platform: NinjaTrader
Broker: Mirus/Zen-FIre
Trading: GC, ES
Posts: 73 since Mar 2010
Thanks Given: 40
Thanks Received: 33

Doh! 4GB? I don't understand. I was under the impression if you opened up a 1-tick chart in NT, that it will save the data into the \Documents\NinjaTrader 7\db\tick folder. I guess that isn't correct then. Right?

If not, then going off of your strategy code, I'd create a binary-writer to register to the contracts and write to disk (via a buffer + some compression algorithms). Then the fun part. We'd create an interface (separate C# app) that would find, decompress and return the data in whatever format your calling program (matlab, r, NT even) would need.

I'd love to help with something like this. I think keeping tick data in a relational database makes no sense whatsoever. Querying the data will take forever and the db size will grow significantly.

Reply With Quote
  #17 (permalink)
 MXASJ 
Asia
 
Experience: Beginner
Platform: NinjaTrader, TOS
Posts: 796 since Jun 2009
Thanks Given: 109
Thanks Received: 800

Where I get lost is if SQL CE is writing binaries to the file system and the DB is merely referencing those binaries... do the binaries themselves count towards the 4 Gig limit or are only the references (which should only be a few bytes) counted.

I'll be honest I'm waaaaaaaaay out of my depth on this one.

Started this thread Reply With Quote
  #18 (permalink)
 phyzfer 
Chicago
 
Experience: Intermediate
Platform: NinjaTrader
Broker: Mirus/Zen-FIre
Trading: GC, ES
Posts: 73 since Mar 2010
Thanks Given: 40
Thanks Received: 33

Personally I don't THINK that NT uses the (SQL CE) database for any market data storage. I think they use it mostly for internal data (account performance, ATM strategies, etc).

I was under the impression that NT saved all Market Data in the Documents\NinjaTrader7\db folder. If you take a look, you'll see a Tick, Minute, Day subfolders that contain binary files (.ntd file extension). How'd these folders get populated?

Personally I'd want NT to save all tick-data for a product to disk whenever any chart is open.

Reply With Quote
  #19 (permalink)
 phyzfer 
Chicago
 
Experience: Intermediate
Platform: NinjaTrader
Broker: Mirus/Zen-FIre
Trading: GC, ES
Posts: 73 since Mar 2010
Thanks Given: 40
Thanks Received: 33

I just got this from the NT's help document:

NinjaTrader Converts Real-Time Data into Historical Data
NinjaTrader stores real-time incoming tick data to your local PC if you have a Chart or Market Analyzer (must have an indicator column added) window open. This data can then be used as historical data. For example, if you open a chart and let it run all day long, the data collected today, will be available as historical data when you open the same chart tomorrow.


I take this to mean that NT is saving tick data on the machine (in the location I specified above) and makes that data available to NT. We'd simply need a way to 'convert' it back and traverse the data. I'd also like to place some sort of backup strategy in place on the machine that is running this data collection.





A colleague of mine suggested I look into this for storing market data:
HDF Group - HDF5

Has anyone used HDF5?

thanks!

Reply With Quote
  #20 (permalink)
 
Trader.Jon's Avatar
 Trader.Jon 
Near the BEuTiFULL Horse Shoe
 
Experience: Beginner
Platform: NinjaTrader
Broker: MBTrading Dukascopy ZenFire
Trading: $EURUSD when it is trending
Posts: 473 since Jul 2009
Thanks Given: 401
Thanks Received: 184


At one time I was talking with a programmer to do this type of project for me: he stopped answering my emails ... cant think why he would do that LOL@me

"" The TREE Data Server captures real-time financial data from one or several datafeed services, archives data in a historical database, and makes both live and archived data available to client applications. The system can be used in real-time charting, an ATS, tick feed simulator, etc. or any situation in which multiple clients need real-time access to the tick stream or archived tick data. In addition, the archived tick stream and tick data are available for offline data analysis and backtesting. ""

Support for the code is one person ... but he does eventually answer email
tree.sourceforge.net | TREE Data Server

TJ

Reply With Quote




Last Updated on July 20, 2012


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts