Millisecond Forums

+x

 

AndrewPapale - 1/3/2022

Has anyone gotten this working with SharePoint or a RedCap database?

A related question, how does Millisecond back up its servers?  (May determine how frequently we pull data, e.g. every month vs every day).

Thanks,
Andrew

It looks like as it currently stands, neither of these options are possible. However, using the HTTP POST option is VERY close. 

Playing around with curl and Microsoft Graph Explorer, it seems like updating from the basic user:password authentication that millisecond provides to allowing a token-based authentication would allow for forwarding data directly to SharePoint (and likely other cloud providers).

Is there a way to request this update? Here is an example for a curl request for running this data forwarding based on the Inquisit, curl, and SharePoint documentation:



Example for the specific SharePoint document library use-case:

Hi Shane,

This does appear feasible. Thanks for laying out the request format, that definitely helps. 

Is the use-case url at the end an example of a SharePoint document library endpoint? If so, presumably the <drive_id> parameter identifies your share, which you would provide as your endpoint, and the <path_to_iqzipfile> parameter represents the relative path within your drive where the data file would be saved. If I've got that right, we would set this to the FileKey, in which case the file/folder structure in your drive would mirror the structure on our servers. 

Also, would you be able to provide a SharePoint drive that we could use to test our implementation? We don't use SharePoint here, and having to set this up might bump the cost from a few hours to a few days. 

Thanks,
Sean

+x

 

seandr - 1/7/2022

Hi Shane,

This does appear feasible. Thanks for laying out the request format, that definitely helps. 

Is the use-case url at the end an example of a SharePoint document library endpoint? If so, presumably the <drive_id> parameter identifies your share, which you would provide as your endpoint, and the <path_to_iqzipfile> parameter represents the relative path within your drive where the data file would be saved. If I've got that right, we would set this to the FileKey, in which case the file/folder structure in your drive would mirror the structure on our servers. 

Also, would you be able to provide a SharePoint drive that we could use to test our implementation? We don't use SharePoint here, and having to set this up might bump the cost from a few hours to a few days. 

Thanks,
Sean

Your understanding <drive_id> and <path_to_iqzipfile> is correct. Thanks for looking into this thus far! I'll go set up a test SharePoint document library and message you the drive_id and an access token. I cannot give you direct access to the SharePoint username and password, but I can confirm the results on my end or work with you to set up an rclone instance of this SharePoint document library.

Going to PM you this now.

Hi Shane,

I've just now published support for SharePoint. I haven't tested it against an actual share, but the protocol is pretty straightforward so this might just work out the box, or at least be very close . 

You can test it out yourself at any time. If there's an error, it will show up in the log for the experiment, which you can see here:
https://myaccount.millisecond.com/experimentlogs

If you get an error, it would be great if you sent the message to support@millisecond.com so we can fix it. 

Thanks,
Sean

Hi Sean,

I set that up, just waiting for either a result or error. Does the API forwarding occur as data is collected or is there some type of schedule for forwarding data?

Also, would it be possible to add one more parameter for specifying where in a drive the data should go? Let's call it <relative_path> for example.

Would something like this be possible?:
'https://graph.microsoft.com/v1.0/drives/<drive_id>/root:/<relative_path>/<path_to_iqzip_file>:/content'

This way, there is some control over where the data is dropped into the SharePoint instead of just being at the root.

Best,
Shane

Forwarding happens immediately when a new data file is uploaded to our server.

If forwarding for a given file fails (e.g. a token expires), the system will re-attempt to forward that file (and any other unforwarded files) the next time any new data from your experiment arrives at the server. We have plans to add UI so that you can manually re-attempt a forward operation, but that's a ways off. 

I've set this up so that it doesn't just dump files into the root folder. The relative path is the same as the path in our data store. For example, the relative path for raw data files is as follows:
<account name>/<experiment name>/<script name>/raw/<datafile name>

We've designed these paths to keep different types of data files separate. If you upload an experiment and run it, you can see the paths yourself by logging into https://myaccount.millisecond.com/datafiles

The SharePoint url as currently designed looks like the following:
https ://graph.microsoft.com/v1.0/drives//root:/<account name>/<experiment name>/<script name>/raw/<datafile name>:/content

That's the intention, anyway. Bugs are always possible, so if your testing shows something different, let me know.

Thanks,
Sean

P.S. If the files aren't showing up in SharePoint and you aren't seeing any errors, I can take a look at look for clues on my end. I'd need the account and experiment names, which you can send to support@millisecond.com or via private message.

+x

 

seandr - 1/11/2022

P.S. If the files aren't showing up in SharePoint and you aren't seeing any errors, I can take a look at look for clues on my end. I'd need the account and experiment names, which you can send to support@millisecond.com or via private message.

Looks like this was not working. Attached is a text file with the report. I double-checked the settings with curl and they looked fine. Is it possible that the token was too long to be properly input?

All errors were the same:

Another item I realized while looking into this more was that the SharePoint tokens have very short lifespans when retrieved from the Graph API Explorer. In inspecting my rclone configuration, I realized that rclone is smart enough to also store a refresh token and expiry of the access token. This makes me think that the PUT request option is not all that feasible. Here is an example of what the rclone configuration looks like:


If someone configured an rclone remote on their local machine, and then maybe copy and paste this as a part of the API Forwarding setup? From there, on the backend it could be as simple as:


That would certainly be more complicated, but rclone could give you access to many cloud resources for data forwarding, listed here: https://rclone.org/#providers

Is your token over 50 characters? If not, the length shouldn't be an issue. I've increased the max size to 100 just in case. 

I'll take a look at SharePoint APIs and see if I can figure out how this is supposed to be done. Whatever method we use will require stable credentials. 

-Sean

+x

 

seandr - 1/15/2022

Is your token over 50 characters? If not, the length shouldn't be an issue. I've increased the max size to 100 just in case. 

I'll take a look at SharePoint APIs and see if I can figure out how this is supposed to be done. Whatever method we use will require stable credentials. 

-Sean

Got a good laugh out of throwing this in python and getting the length. Looks like it's 2366 characters long. Thanks for looking into the SharePoint API. Feel free to reach out if I can be of any help.

Best,
Shane