Millisecond Forums

+x

 

loganap - 9/20/2021

Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 

+x

 

seandr - 9/20/2021

+x

 

loganap - 9/20/2021

Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 

Thank you, Sean. That might be just what I need! Can you point me to the documentation of this feature? I must have overlooked it in my searches.

+x

 

loganap - 9/20/2021

+x

 

seandr - 9/20/2021

+x

 

loganap - 9/20/2021

Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 

Thank you, Sean. That might be just what I need! Can you point me to the documentation of this feature? I must have overlooked it in my searches.

Ah, found it. In case anyone needs it: https://www.millisecond.com/support/docs/v5/html/viewer.htm#articles/dataweb.htm
This can be done via the <data> tag in your experiment configuration.

Thanks again for pointing me in the right direction.

+x

 

loganap - 9/20/2021

+x

 

loganap - 9/20/2021

+x

 

seandr - 9/20/2021

+x

 

loganap - 9/20/2021

Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 

Thank you, Sean. That might be just what I need! Can you point me to the documentation of this feature? I must have overlooked it in my searches.

Ah, found it. In case anyone needs it: https://www.millisecond.com/support/docs/v5/html/viewer.htm#articles/dataweb.htm
This can be done via the <data> tag in your experiment configuration.

Thanks again for pointing me in the right direction.

There's another, somewhat more convenient and more flexible option than using the <data> element. In your web experiment's settings, under "Advanced Settings", you can enable the data forwarding option, select the forwarding protocol (options are: Multipart Form POST, HTTP/REST POST or PUT, and AWS S3 forwarding), enter your endpoint's address, as well as the access credentials required by the endpoint (if any).



Two major advantages of this: You don't need to change anything in your scripts, they can use the existing / default <data options, and you can optionally keep a copy on the Millisecond servers as a backup.

+x

 

Dave - 9/20/2021

+x

 

loganap - 9/20/2021

+x

 

loganap - 9/20/2021

+x

 

seandr - 9/20/2021

+x

 

loganap - 9/20/2021

Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 

Thank you, Sean. That might be just what I need! Can you point me to the documentation of this feature? I must have overlooked it in my searches.

Ah, found it. In case anyone needs it: https://www.millisecond.com/support/docs/v5/html/viewer.htm#articles/dataweb.htm
This can be done via the <data> tag in your experiment configuration.

Thanks again for pointing me in the right direction.

There's another, somewhat more convenient and more flexible option than using the <data> element. In your web experiment's settings, under "Advanced Settings", you can enable the data forwarding option, select the forwarding protocol (options are: Multipart Form POST, HTTP/REST POST or PUT, and AWS S3 forwarding), enter your endpoint's address, as well as the access credentials required by the endpoint (if any).



Two major advantages of this: You don't need to change anything in your scripts, they can use the existing / default <data options, and you can optionally keep a copy on the Millisecond servers as a backup.

Can anyone point to documentation on this feature? I'd like to configure our server to receive these requests but I'd like to see how the POST data will be sent. I didn't see it in the online docs. Thanks!

+x

 

loganap - 9/24/2021

+x

 

Dave - 9/20/2021

+x

 

loganap - 9/20/2021

+x

 

loganap - 9/20/2021

+x

 

seandr - 9/20/2021

+x

 

loganap - 9/20/2021

Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 

Thank you, Sean. That might be just what I need! Can you point me to the documentation of this feature? I must have overlooked it in my searches.

Ah, found it. In case anyone needs it: https://www.millisecond.com/support/docs/v5/html/viewer.htm#articles/dataweb.htm
This can be done via the <data> tag in your experiment configuration.

Thanks again for pointing me in the right direction.

There's another, somewhat more convenient and more flexible option than using the <data> element. In your web experiment's settings, under "Advanced Settings", you can enable the data forwarding option, select the forwarding protocol (options are: Multipart Form POST, HTTP/REST POST or PUT, and AWS S3 forwarding), enter your endpoint's address, as well as the access credentials required by the endpoint (if any).



Two major advantages of this: You don't need to change anything in your scripts, they can use the existing / default <data options, and you can optionally keep a copy on the Millisecond servers as a backup.

Can anyone point to documentation on this feature? I'd like to configure our server to receive these requests but I'd like to see how the POST data will be sent. I didn't see it in the online docs. Thanks!

I'll outline it here. Five metadata fields are forwarded as query parameters to the endpoint. Those are:

ScriptPath (the script file name)
GroupId (the group number)
SubjectId (the subject ID)
SessionId (the session ID)
FileKey (the data file path & name)

The POST message's body is the actual data file.

Below screenshot should illustrate:



In essence, this is also what you'll find outlined here https://www.millisecond.com/products/inquisit6/webhosting.aspx under "Hosting Data Upload Service."

I've just updated the help topic on this, which was woefully out of date. It expands a bit on Dave's reply.

You can find details of various protocol options here:
https://www.millisecond.com/support/docs/v6/html/articles/dataweb.htm

+x

 

seandr - 9/24/2021

I've just updated the help topic on this, which was woefully out of date. It expands a bit on Dave's reply.

You can find details of various protocol options here:
https://www.millisecond.com/support/docs/v6/html/articles/dataweb.htm

Perfect, thanks

Has anyone gotten this working with SharePoint or a RedCap database?

A related question, how does Millisecond back up its servers?  (May determine how frequently we pull data, e.g. every month vs every day).

Thanks,
Andrew