Downloading and uploading binary files with VCF Orchestrator
Saving and sending binary data over HTTP(S) with VCF Orchestrator
Table of Contents
Introduction
VCF Orchestrator HTTP REST plugin allows to download and upload data via HTTP protocol. This feature is crucial to integrate 3rd party systems via API.
REST is a common choice of these API implementations, using mostly text (often JSON) based communication.
Hovever, transferring binary data (files) is sometimes required. Let's explore a possible way to do this within Orchestrator.
Limitations of the HTTP REST plugin
When executing requests via the plugin, the RESTResponse object has the following attributes, functions:
As the name suggests, contentAsString provides the request response as string.
Let's try to download an image and see the first 16 bytes:
1$ curl -sk https://httpbin.org/image/jpeg | od -N 16 -t u1
20000000 255 216 255 224 0 16 74 70 73 70 0 1 1 2 0 28
30000020
Let's print the first 16 bytes of the same file downloaded by vRO, native JavaScript:
1var restHostUrl = "https://httpbin.org";
2
3// importing certificate to cert store
4var ld = Config.getKeystores().getImportCAFromUrlAction();
5ld.setCertificateAlias("");
6var model = ld.getModel();
7model.value = restHostUrl;
8var importresult = ld.execute();
9
10// build a dynamic host
11var host = RESTHostManager.createHost("dynamicREST");
12var restHost = RESTHostManager.createTransientHostFrom(host);
13restHost.hostVerification = false;
14restHost.url = restHostUrl;
15
16// make request
17var request = restHost.createRequest("GET", encodeURI("/image/jpeg"));
18var response = request.execute();
19var content = response.contentAsString;
20
21var a = [];
22for (i = 0; i < 16; i++) {
23 var code = content.charCodeAt(i);
24 a.push(code);
25}
26System.log(a);
27System.log(content.substr(0,16));
Here are the logs:
The first four bytes are unfortunately converted to Unicode replacement character code 65533 (0xFFFD): �
This conversion happens automatically when the plugin loads the data into a String variable. This modifies the content downloaded, so we cannot get the original data.
Workaround: use Python
Addinional languages are available in Orchestrator if the license applied allows them (Orchestrator embedded in VCF Automation). Let's use Python do handle the network I/O.
File download
We download a binary file using the urllib module (available on the appliance by default). The contents of the file will be base64 encoded to avoid encoding issues with String representation of Orchestrator.
1from urllib import request
2import ssl
3import base64
4
5def handler(context, inputs):
6
7 r = request.Request("https://httpbin.org/image/jpeg")
8 # disable SSL verification
9 response = request.urlopen(r, context=ssl.SSLContext())
10
11 return {
12 "filename": "image.jpg",
13 "mimetype": response.getheader('Content-Type'),
14 "filecontent": base64.b64encode(response.read()).decode('ascii')
15 }
We use the ByteBuffer type to load the data into a MimeAttachment object. The construstor of ByteBuffer allows initialization by a base64 String.
Then we print the first 20 characters and save it into the resource element folder specified by the input variable of type ResourceElementCategory.
1var buffer = new ByteBuffer(filecontent);
2var attachment = new MimeAttachment();
3attachment.name = filename;
4attachment.mimeType = mimetype;
5attachment.buffer = buffer;
6
7System.log("filename: " + attachment.name);
8System.log("mimetype: " + attachment.mimeType);
9System.log("content : " + attachment.content.substr(0, 20) + "..."); // first 20 characters
10
11var element;
12for each (e in folder.resourceElements) {
13 if (e.name == filename) {
14 element = e;
15 break;
16 }
17}
18
19if (element) // overwrite
20 element.setContentFromMimeAttachment(attachment);
21else // create
22 Server.createResourceElement(folder.path, filename, attachment);
23
24System.log(folder.path + "/" + filename + " saved.");
The input form:
Webserver response:
The downloaded file saved as a resource element:
File upload
I've used this method in an earlier blog post "vRO8 Import Workflow" related to the Orchestrator API. If you are interested in the network traffic, please check this link for further details wrt. multipart/form-data Content-Type.
I reused Doug Hellmann's implementation of Multipart Content type available at Python 3 Module of the Week:
1import io
2import mimetypes
3from urllib import request
4import uuid
5import ssl
6import base64
7
8def handler(context, inputs):
9
10 # Create the form with simple fields
11 form = MultiPartForm()
12 # Add the file
13 form.add_file(
14 'file', inputs["file"].split(',', 1)[0][1:], # filename: in first line
15 fileHandle=io.BytesIO(base64.b64decode(inputs["file"].split('\n',1)[1]))) # content in the second
16
17 # Build the request, including the byte-string
18 # for the data to be posted.
19 data = bytes(form)
20
21 r = request.Request('https://api.escuelajs.co/api/v1/files/upload', data=data)
22 r.add_header('Content-type', form.get_content_type())
23 r.add_header('Content-length', len(data))
24
25 print('SERVER RESPONSE:')
26 # disable SSL verification
27 print(request.urlopen(r, context=ssl.SSLContext()).read().decode('utf-8'))
28
29class MultiPartForm:
30 ...
The input form:
Webserver response:
Uploaded file:
Direct copy between APIs
Sometimes we do not need to store a downloaded file, but upload to a different location. This case we can avoid JavaScript completely, handling the whole process within a single Python code. We can use the io.BytesIO in-memory binary stream object to store the data temporarily. The combined code:
1import io
2import mimetypes
3from urllib import request
4import uuid
5import ssl
6import base64
7
8def handler(context, inputs):
9
10 r = request.Request("https://httpbin.org/image/jpeg")
11 # disable SSL verification
12 response = request.urlopen(r, context=ssl.SSLContext())
13
14 # Create the form with simple fields
15 form = MultiPartForm()
16 # Add the file
17 form.add_file(
18 "file", "image.jpg", # filename
19 fileHandle=io.BytesIO(response.read())) # content
20
21 # Build the request, including the byte-string
22 # for the data to be posted.
23 data = bytes(form)
24
25 r = request.Request('https://api.escuelajs.co/api/v1/files/upload', data=data)
26 r.add_header('Content-type', form.get_content_type())
27 r.add_header('Content-length', len(data))
28
29 print('SERVER RESPONSE:')
30 # disable SSL verification
31 print(request.urlopen(r, context=ssl.SSLContext()).read().decode('utf-8'))
32
33class MultiPartForm:
34...
Workflow logs:
Uploaded file:
Final thoughts
Please note that the above methods do not scale well for large files. If you need to work with big binary files, leverage an external server to do the weight lifting and use Orchestrator to run commands on this external server.
You can download the com.test.binaryfiles package containing the workflow from GitHub: https://github.com/kuklis/vro8-packages