Downloading and uploading binary files with VCF Orchestrator

Share on:

Saving and sending binary data over HTTP(S) with VCF Orchestrator

Introduction

VCF Orchestrator HTTP REST plugin allows to download and upload data via HTTP protocol. This feature is crucial to integrate 3rd party systems via API.
REST is a common choice of these API implementations, using mostly text (often JSON) based communication.

Hovever, transferring binary data (files) is sometimes required. Let's explore a possible way to do this within Orchestrator.

Limitations of the HTTP REST plugin

When executing requests via the plugin, the RESTResponse object has the following attributes, functions:

As the name suggests, contentAsString provides the request response as string.

Let's try to download an image and see the first 16 bytes:

1$ curl -sk https://httpbin.org/image/jpeg | od -N 16 -t u1
20000000 255 216 255 224   0  16  74  70  73  70   0   1   1   2   0  28
30000020

Let's print the first 16 bytes of the same file downloaded by vRO, native JavaScript:

 1var restHostUrl = "https://httpbin.org";
 2
 3// importing certificate to cert store
 4var ld = Config.getKeystores().getImportCAFromUrlAction();
 5ld.setCertificateAlias("");
 6var model = ld.getModel();
 7model.value = restHostUrl;
 8var importresult = ld.execute();
 9
10// build a dynamic host
11var host = RESTHostManager.createHost("dynamicREST");
12var restHost = RESTHostManager.createTransientHostFrom(host);
13restHost.hostVerification = false;
14restHost.url = restHostUrl;
15
16// make request
17var request = restHost.createRequest("GET", encodeURI("/image/jpeg"));
18var response = request.execute();
19var content = response.contentAsString;
20
21var a = [];
22for (i = 0; i < 16; i++) {
23    var code = content.charCodeAt(i);
24    a.push(code);
25}
26System.log(a);
27System.log(content.substr(0,16));

Here are the logs:

The first four bytes are unfortunately converted to Unicode replacement character code 65533 (0xFFFD): �
This conversion happens automatically when the plugin loads the data into a String variable. This modifies the content downloaded, so we cannot get the original data.

Workaround: use Python

Addinional languages are available in Orchestrator if the license applied allows them (Orchestrator embedded in VCF Automation). Let's use Python do handle the network I/O.

File download

We download a binary file using the urllib module (available on the appliance by default). The contents of the file will be base64 encoded to avoid encoding issues with String representation of Orchestrator.

 1from urllib import request
 2import ssl
 3import base64
 4
 5def handler(context, inputs):
 6
 7    r = request.Request("https://httpbin.org/image/jpeg")
 8    # disable SSL verification
 9    response = request.urlopen(r, context=ssl.SSLContext())
10    
11    return {
12        "filename": "image.jpg", 
13        "mimetype": response.getheader('Content-Type'),
14        "filecontent": base64.b64encode(response.read()).decode('ascii')
15    }

We use the ByteBuffer type to load the data into a MimeAttachment object. The construstor of ByteBuffer allows initialization by a base64 String.
Then we print the first 20 characters and save it into the resource element folder specified by the input variable of type ResourceElementCategory.

 1var buffer = new ByteBuffer(filecontent);
 2var attachment = new MimeAttachment();
 3attachment.name = filename;
 4attachment.mimeType = mimetype;
 5attachment.buffer = buffer;
 6
 7System.log("filename: " + attachment.name);
 8System.log("mimetype: " + attachment.mimeType);
 9System.log("content : " + attachment.content.substr(0, 20) + "..."); // first 20 characters
10
11var element;
12for each (e in folder.resourceElements) {
13    if (e.name == filename) {
14        element = e;
15        break;
16    }
17}
18
19if (element) // overwrite
20    element.setContentFromMimeAttachment(attachment);
21else         // create
22    Server.createResourceElement(folder.path, filename, attachment);
23
24System.log(folder.path + "/" + filename + " saved.");

The input form:

Webserver response:

The downloaded file saved as a resource element:

File upload

I've used this method in an earlier blog post "vRO8 Import Workflow" related to the Orchestrator API. If you are interested in the network traffic, please check this link for further details wrt. multipart/form-data Content-Type.

I reused Doug Hellmann's implementation of Multipart Content type available at Python 3 Module of the Week:

 1import io
 2import mimetypes
 3from urllib import request
 4import uuid
 5import ssl
 6import base64
 7
 8def handler(context, inputs):
 9
10    # Create the form with simple fields
11    form = MultiPartForm()
12    # Add the file
13    form.add_file(
14        'file', inputs["file"].split(',', 1)[0][1:],                              # filename: in first line
15        fileHandle=io.BytesIO(base64.b64decode(inputs["file"].split('\n',1)[1]))) # content in the second
16
17    # Build the request, including the byte-string
18    # for the data to be posted.
19    data = bytes(form)
20
21    r = request.Request('https://api.escuelajs.co/api/v1/files/upload', data=data)
22    r.add_header('Content-type', form.get_content_type())
23    r.add_header('Content-length', len(data))
24 
25    print('SERVER RESPONSE:')
26    # disable SSL verification
27    print(request.urlopen(r, context=ssl.SSLContext()).read().decode('utf-8'))
28
29class MultiPartForm:
30    ...

The input form:

Webserver response:

Uploaded file:

Direct copy between APIs

Sometimes we do not need to store a downloaded file, but upload to a different location. This case we can avoid JavaScript completely, handling the whole process within a single Python code. We can use the io.BytesIO in-memory binary stream object to store the data temporarily. The combined code:

 1import io
 2import mimetypes
 3from urllib import request
 4import uuid
 5import ssl
 6import base64
 7
 8def handler(context, inputs):
 9
10    r = request.Request("https://httpbin.org/image/jpeg")
11    # disable SSL verification
12    response = request.urlopen(r, context=ssl.SSLContext())
13
14    # Create the form with simple fields
15    form = MultiPartForm()
16    # Add the file
17    form.add_file(
18        "file", "image.jpg",                    # filename
19        fileHandle=io.BytesIO(response.read())) # content
20
21    # Build the request, including the byte-string
22    # for the data to be posted.
23    data = bytes(form)
24
25    r = request.Request('https://api.escuelajs.co/api/v1/files/upload', data=data)
26    r.add_header('Content-type', form.get_content_type())
27    r.add_header('Content-length', len(data))
28 
29    print('SERVER RESPONSE:')
30    # disable SSL verification
31    print(request.urlopen(r, context=ssl.SSLContext()).read().decode('utf-8'))
32
33class MultiPartForm:
34...

Workflow logs:

Uploaded file:

Final thoughts

Please note that the above methods do not scale well for large files. If you need to work with big binary files, leverage an external server to do the weight lifting and use Orchestrator to run commands on this external server.

You can download the com.test.binaryfiles package containing the workflow from GitHub: https://github.com/kuklis/vro8-packages