Uploading data to, and retrieving data from, Timetric¶
We’ll show you, firstly, how to get data into the system, and secondly, how to get it out again. (Before you get started, you’ll need to set yourself up with authentication credentials.)
Inserting data¶
Adding pre-existing data¶
Perhaps you have an existing data series you want to upload to the platform. This might be a public data set you’ve come across, or it might be data you’ve generated yourself.
Currently, the first thing you need to do is convert it into CSV that Timetric understands.
- you need one CSV file for each individual series.
- each CSV file must have two columns
- the timestamps and values must be encoded correctly, as shown elsewhere.
An example:
2003-04-06T00:00:00, 97755.0
2004-04-06T00:00:00, 105467.0
2005-04-06T00:00:00, 105953.0
2006-04-06T00:00:00, 113192.0
2007-04-06T00:00:00, 110526.0
2008-04-06T00:00:00, 103999.0
To upload this to Timetric, you need to POST the data to /create/, the POSTed file being called csv. You’ll also need to add a few parameters. At a minimum, you’ll want a caption for your dataset, a short title, and probably some tags. You might also want some geographical data attached. (The full list of parameters can be found here).
When POSTing a file, there are two ways to encode the data; depending on your HTTP library, one or the other may be easiest.
Firstly, you can encode the file data and parameters in multipart/form-data format. If your HTTP library will construct this for you automatically, then this will be easiest; otherwise it’s rather awkward to construct correctly. (but see for example this Python snippet).
Secondly, you can encode any parameters in the URL as query parameters, and upload the file as the body of the HTTP request. This is much simpler to construct if your HTTP library doesn’t provide multipart capability, so this is what we’ll document here.
So, a simple upload of the above series, to be entitled “Some interesting data”, would take these parameters:
title: "Some interesting data"
caption: "Much longer description of this data"
and require access to the data itself in the body of the request, which would look like:
https://timetric.com/create/?title=Some%20interesting%20data;caption=Much%20longer%20description%20of%20this%20data
POST /create/ HTTP/1.1
Host: timetric.com
Content-Type: text/csv
Content-Length: 185
2003-04-06T00:00:00, 97755.0
2004-04-06T00:00:00, 105467.0
2005-04-06T00:00:00, 105953.0
2006-04-06T00:00:00, 113192.0
2007-04-06T00:00:00, 110526.0
2008-04-06T00:00:00, 103999.0
Note that both the Content-Type and Content-Length headers are required.
Note
Content-Length should be the length of the body in bytes; take care when counting newline characters.
The response will look something like this:
HTTP/1.x 303 SEE OTHER
Date: Wed, 07 Jan 2009 16:35:09 GMT
Server: Apache/2.2.8 (Ubuntu) mod_wsgi/2.3 Python/2.5.2
Location: http://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/
Content-Length: 0
Content-Type: text/html; charset=utf-8
As you can see, the URL of the new series is shown in the HTTP Location header. That URL is now the canonical location for your uploaded data.
Note that processing of uploaded data is asynchronous. A resource for the series is created immediately, and a GET on the URL will succeed. However, the data may not be there immediately; depending on the size of the dataset uploaded, it may be several seconds before the data is available.
Creating a new series for later updating¶
If want to use Timetric as a data-capture platform, you need a URL for your data — so, first of all, you need to create an empty series.
You can use the same interface as above, except that you don’t need to supply any initial data. Let’s register a new series, except this time with tags and giving it a geographical location. In this case, we could pass the data in as query parameters to the URL, as above, or alternatively we can put them in the body of the message using the x-www-form-urlencoded mimetype, which we illustrate here.
https://timetric.com/create/
POST /create/ HTTP/1.1
Host: timetric.com
Content-Type: application/x-www-form-urlencoded
title=Empty&caption=Empty&tags=sensor%20datacapture&latitude=54.0%longitude=-0.3
Again, the response will look something like the following:
HTTP/1.x 303 SEE OTHER
Date: Wed, 07 Jan 2009 16:35:09 GMT
Server: Apache/2.2.8 (Ubuntu) mod_wsgi/2.3 Python/2.5.2
Location: http://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/
Content-Length: 0
Content-Type: text/html; charset=utf-8
showing the URL of the new, empty, series, in the Location header.
Updating an existing series¶
You might want to update an existing series if:
- It’s a dataset you’re mirroring, and some new data has become available (perhaps these are data reported annually by an organization you’re interested in.)
- It’s a dataset you generate yourself, and you’ve just collated a new set of values (perhaps you collate these quarterly, and you want to upload the last three months’ worth of data.)
- You’re using Timetric as a data capture backend, and you want to update the data series every time you get a new value from your data generator — perhaps you have an environmental sensor system uploading data every 5 minutes.
The first two cases are treated in the same way.
Updating a dataset¶
You need to have your data in the Timetric CSV format. Your CSV file can contain either only the new data, or a mixture of old and new data (sometimes it’s easier not to have to filter out old data). The server will behave identically either way.
Note
You can’t change existing data points in this way, though. Attempts to do so are an error.
Now, simply POST the new data set to the URL of the series, with a filename of csv. Again, you can send the file as multipart/form-data, if your HTTP library makes it easy for you, but otherwise it’s simpler to send the data in the body of the request, with a Content-Type of text/csv/.
https://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/
POST /series/dQq_dbUOTZmCq6bbNCCMJA/ HTTP/1.1
Host: timetric.com
Content-Type: text/csv
Content-Length: 185
2003-04-06T00:00:00, 97755.0
2004-04-06T00:00:00, 105467.0
2005-04-06T00:00:00, 105953.0
2006-04-06T00:00:00, 113192.0
2007-04-06T00:00:00, 110526.0
2008-04-06T00:00:00, 103999.0
The server should respond with a 204:
HTTP/1.x 204
Date: Wed, 07 Jan 2009 16:35:09 GMT
Server: Apache/2.2.8 (Ubuntu) mod_wsgi/2.3 Python/2.5.2
Location: http://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/
Updating a value¶
If you need just to update the value of the series, there are three cases to think about.
Updating the series with a new value at a known timestamp — perhaps you have a sensor with its own timing device attached and you need to record measurement time precisely.
In this case, simply follow the recipe for updating a dataset, but use a CSV file with only one entry.
Updating the series with a new value, timestamped with “now” — perhaps you have a very lightweight sensor with no timing device.
Rather than updating to a new absolute value, you can simply increment or decrement the existing value. (Perhaps your sensor only registers changes in its environment: “three more people walked through the door”, rather than the absolute value of the quantity you care about: “ten people are in the room”.)
For the second case, you need to POST to the URL, but instead of attaching a CSV file, send a value parameter.
https://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/
POST /series/dQq_dbUOTZmCq6bbNCCMJA/ HTTP/1.1
Host: timetric.com
Content-Type: application/x-www-form-urlencoded
value=107632.0
For the third case, you need to do almost the same thing — but instead of using the value parameter, use the increment parameter with the change in value (which can be negative to reduce a total).
https://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/
POST /series/dQq_dbUOTZmCq6bbNCCMJA/ HTTP/1.1
Host: timetric.com
Content-Type: application/x-www-form-urlencoded
increment=-3.0
In both cases, the server should respond with a 204.
HTTP/1.x 204
Date: Wed, 07 Jan 2009 16:35:09 GMT
Server: Apache/2.2.8 (Ubuntu) mod_wsgi/2.3 Python/2.5.2
Location: http://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/
Retrieving data¶
Getting the whole series¶
The data for the whole series can be retrieved as a CSV file; there are two slightly different formats available:
http://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/csv/unix.csv
http://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/csv/iso.csv
In each case, the URL will return a two-column CSV file containing the data for the whole series.
- In unix.csv, the timestamps will be encoded as numbers representing unix timestamps (seconds until/since 1st January 1970).
- In iso.csv, the timestamps will be encoded as W3C date-times.
(See the CSV format documentation.)
Getting the latest value¶
To get at the latest value of a given series, you simply need to retrieve the value/ of its URL, which you can ask for as json/:
http://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/value/json/
GET /series/dQq_dbUOTZmCq6bbNCCMJA/value/json/ HTTP/1.1
Host: timetric.com
and the server will respond with a JSON representation of the latest value and its timestamp:
HTTP/1.x 200 OK
Date: Wed, 07 Jan 2009 18:09:09 GMT
Server: Apache/2.2.8 (Ubuntu) mod_wsgi/2.3 Python/2.5.2
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/json
{"timestamp": 1207440000.0, "value": 103999.0}
Addressing individual values¶
Individual points within a series can be referenced with the following URL:
http://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/__TIMESTAMP__/
where __TIMESTAMP__ is the timestamp of the point in question.
This timestamp can be specified in two ways; as an ISO date-time string, or as a Unix timestamp. In either case, the result of a GET to such a URL is a CSV document containing a single (timestamp, value) pair, corresponding to the point in that series at the requested timestamp, or the last point defined prior to it.
http://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/2009-03-16T11:00:00/
GET /series/dQq_dbUOTZmCq6bbNCCMJA/2009-03-16T11:00:00 HTTP/1.1
Host: timetric.com
and the server will respond with a CSV representation of the requested point.
HTTP/1.x 200 OK
Date: Wed, 07 Jan 2009 18:09:09 GMT
Server: Apache/2.2.8 (Ubuntu) mod_wsgi/2.3 Python/2.5.2
Content-Type: text/csv
2009-03-16T11:00:00, 6889.0
If the timestamp is given in the URL as an ISO datetime string, the response will be returned with a datetime string; if it is given as a Unix timestamp, it will be returned as a Unix timestamp.
Deleting data¶
Sadly, sometimes you’ll need to remove data. Perhaps you uploaded all your internal sales projections and even though they’re marked private, you’d rather not have them on someone else’s server!
Simply issue a DELETE to the series URL.
https://timetric.com/series/dQq_dbUOTZmCq6bbNCCMJA/
DELETE /series/dQq_dbUOTZmCq6bbNCCMJA/ HTTP/1.1
Host: timetric.com
The server will respond with a 204
HTTP/1.x 204
Date: Wed, 07 Jan 2009 16:35:09 GMT
Server: Apache/2.2.8 (Ubuntu) mod_wsgi/2.3 Python/2.5.2