Google Cloud
Load to variable from GCS
Overview
This BLOCK loads the contents of a file stored into Google Cloud Storage (GCS) into a variable.
For example, assume a text file with the following contents is stored in GCS at gs://magellan-sample/files.txt:
sample1.png
sample2.png
Using this BLOCK with this file would set a variable to the following data:
{
"gcs_url": "gs://magellan-sample/files.txt",
"timestamp": 1480986550.0,
"content": "sample1.png\nsample2.png"
}
As shown above, the variable contains the contents of the file along with other incidental information as JSON format data. The "content" portion contains the file’s contents as "sample1.png\nsample2.png" (\n represents a line break).
You can select the file format of the file from among (none), YAML, JSON, Newline delimited JSON, Binary (Base64), and CSV.
| File format | Explanation |
|---|---|
| (none) |
"content" is set as the contents of the file just as they are. |
| YAML/JSON |
"content" is set as the results of parsing the contents of the file according to the selected format. For example, assume that the file contains the following YAML format text: Basic: - Start of Flow - Parallel branch - Slack notice - Output to log - Construct object - End of Flow API: - HTTP GET - HTTP POST - HTTP PUT - HTTP DELETE After processing this file with this BLOCK, "content" would contain the following:
"content": {
"Basic": [
"Start of Flow",
"Parallel branch",
"Slack notice",
"Output to log",
"Construct object",
"End of Flow"
],
"API": [
"HTTP GET",
"HTTP POST",
"HTTP PUT",
"HTTP DELETE"
]
}
|
| Newline delimited JSON |
"content" is set as the results of converting newline delimited JSON into an array. Each line of the JSON becomes one element of the array. For example, assume that the file contains the following newline delimited JSON text:
{"name1": "value1", "name2": "value2"}
{"name1": "value3", "name2": "value4"}
After processing this file with this BLOCK, "content" would contain the following:
"content": [
{
"name1": "value1",
"name2": "value2"
},
{
"name1": "value3",
"name2": "value4"
}
]
|
| Binary (Base64) |
"content" is set as Base64 encoded contents of the file. For example, "content" could contain something like the following: "content": "ZkxhQwAA……qqA55Q==" |
| CSV |
"content" is set as the results of analyzing the contents of the file as CSV format and converting them into an array of objects. All fields in the CSV file are handled as strings during the conversion.。 CSV files are assumed to hold a header line, as shown below. The header line must have the same number of fields as the other lines, and its fields will be used as the list of field names. field_name_1,field_name_2,field_name_3 aaa,bbb,ccc xxx,yyy,zzz After processing this CSV file with this BLOCK, "content" would contain the following array of objects:
"content": [
{
"field_name_1": "aaa",
"field_name_2": "bbb",
"field_name_3": "ccc"
},
{
"field_name_1": "xxx",
"field_name_2": "yyy",
"field_name_3": "zzz"
}
]
|
Properties
| Property | Explanation | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| BLOCK name | Configure the name displayed on this BLOCK. | ||||||||
| GCP service account | Select the GCP service account to use with this BLOCK. | ||||||||
| Source file GCS URL | Designate the GCS URL of the file whose contents will be loading into the variable. Example: gs://magellan-sample/sample.yaml | ||||||||
| File format | Select the format of the GCS file as (none) , YAML, JSON, Newline delimited JSON, Binary (Base64), or CSV. | ||||||||
| BLOCK memos | Make any comments about this BLOCK. | ||||||||
| CSV delimiter character |
When the file format property is set to CSV, select the delimiter character used in the CSV file as Comma (,), Tab, Pipe (|), or Other. If you select Other, enter the delimiter character used into the adjacent field. |
||||||||
| Number of skipped rows |
When the file format property is set to CSV, configure the number of lead rows to skip. |
||||||||
| Designate quotation marks |
When the file format property is set to CSV, designate the character used for quotation marks. |
||||||||
| Results variable |
Designate the variable that will store the contents of the GCS file. The data stored into the variable is formatted as follows:
{
"gcs_url": "...",
"timestamp": ...,
"content": ...
}
|