Skip to main content

Lesson 6 - Resources

Adding a wordlist in the Runefile#

In lesson-2, we look at how to manually add labels in the args property of the hotg-ai/proc-blocks@label proc-block. This will be a cumbersome process if we have a large number of labels (e.g. the Landmark model has 100k labels).

Instead of manually adding all those labels in the Runefile, we can provide a path to the txt or csv files using a resource.

Resources are declared under the Runefile's resources property.

version: 1
image: runicos/base
pipeline:
image:
capability: IMAGE
outputs:
- type: U8
dimensions: [1, 96, 96, 1]
args:
pixel-format: "@PixelFormat::GrayScale"
height: 96
width: 96
person_detection:
model: "./person_detection.tflite"
inputs:
- image
outputs:
- type: U8
dimensions: [1, 1, 1, 3]
most_confident_index:
proc-block: "hotg-ai/proc-blocks#most_confident_indices"
inputs:
- person_detection
outputs:
- type: U32
dimensions: [1]
args:
count: 1
label:
proc-block: "hotg-ai/proc-blocks#label"
inputs:
- most_confident_index
outputs:
- type: UTF8
dimensions: [1]
args:
wordlist: $WORD_LIST # <-------------
serial:
out: SERIAL
inputs:
- label
resources: # <---------------
WORD_LIST:
path: ./label.txt
type: string

Proc-blocks#

The processing block is used to process your data by transforming it from one form to another. We have a list of them over here. You can also create your own proc-block if you need one, or let us know we can help you make one.

1. argmax#

This proc-block returns indices of the max element of the array.

image: runicos/base
version: 1
pipeline:
random_input:
capability: RAND
outputs:
- type: f32
dimensions: [5]
argmax:
proc-block: "hotg-ai/proc-blocks#argmax"
inputs:
- random_input
outputs:
- type: u32
dimensions: [1]
serial:
out: SERIAL
inputs:
- random_input
- argmax

E.g. I got this output after running rune build from the above Runefile. As we can see 2nd index has largest value.

random_input: {
"type_name":"f32",
"channel":1,
"elements":[-5.7317625e 28, -1.5026809e14, 3.0563458e26, 8.577695e22, 5.0970436e-17],
"dimensions":[5]
}
argmax: {
"type_name":"u32",
"channel":1,
"elements":[2],
"dimensions":[1]
}

It takes only type f32 tensor as input and give back u32 tensor as output. This proc-block get used in the BERT rune.

2. audio_float_conversion#

Many audio models take in an input of f32. This proc-block convert our input from an i16 data type to a floating-point value.

This proc-block used in the YAMNET rune. You can find more details on this in our article.

3. fft#

This proc-block converts a signal from its original domain (often time or space) to a representation in the frequency domain.

This proc-block is used in the microspeech rune. You can find more detail in our article.

4. image-normalization#

We saw this proc-block in lesson-3. It transform our u8 type to f32 and normalize the image matrix in range [0, 1].

5. label#

We saw this proc-block in lesson-2. A proc block which, when given a set of indices, will return their associated labels.

6. modulo#

As the same suggest, it returns the remainder of a division, after one number is divided by another.

This proc-block is used in the sine rune.

7. most_confident_indices.#

We saw this proc-block in lesson-2. A proc block which, when given a list of confidences, will return the indices of the top N most confident values.

8. noise-filtering#

This proc-block perform a couple of functions:

  • Reduces noise within each frequency bin (channel)
  • Applies a gain control algorithm to each frequency bin (channel)
  • Applies log2 function and scales the output.

This proc-block is used in the microspeech rune. You can find more detail in our article.

9. normalize#

This proc-block normalize the input to the range [0, 1].

This proc-block is used in the gesture rune.

10. parse#

A proc block which can parse a string to numbers. This proc-block could be helpful in doing non-ML tasks.

image: runicos/base
version: 1
pipeline:
number_list:
capability: RAW
outputs:
- type: u8
dimensions: [50] # ----> keep this dimension a bit larger rune will automatically adjust it.
utf8_decode:
proc-block: "hotg-ai/proc-blocks#utf8_decode"
inputs:
- number_list
outputs:
- type: utf8
dimensions: [50] # ----> keep this dimension a bit larger rune will automatically adjust it.
parse:
proc-block: "hotg-ai/proc-blocks#parse"
inputs:
- utf8_decode
outputs:
- type: u32
dimensions: [5]
word_list:
proc-block: "hotg-ai/proc-blocks#label"
inputs:
- parse
outputs:
- type: utf8
dimensions: [5]
args:
labels:
- zero
- one
- two
- three
- four
- five
- six
- seven
- eight
- nine
- ten
serial:
out: SERIAL
inputs:
- word_list

After building a rune using above Runefile run rune run test.rune --raw input.txt, where input .txt contains 1 2 3 4 5. You will get this as output.

{"type_name":"utf8","channel":1,"elements":["one","two","three","four","five"],"dimensions":[5]}

P.S. Keep capability dimension a bit a bit larger rune will automatically adjust it.

11. segment_output#

This proc-block is useful in image segmentation. A proc-block which takes a rank 4 tensor as input, whose dimension is of this form [1, x, y, z]. It will return:

  • a 2-d tensor after performing argmax along the axis-3 of the tensor
  • a 1-d tensor which a set of all the number present in the above 2-d tensor

We use this in the deep_lab rune.

12. softmax#

A proc-block which apply softmax function over the tensor list.

13. tokenizers#

This proc-block is helpful in the NLP models. Tokenization is a way of separating a piece of text into smaller units called tokens (e.g. bert tokenizer).

tokens: ['[CLS]', 'this', 'is', 'a', 'nice', 'sentence', '.', '[SEP]']
input_ids: [101, 2023, 2003, 1037, 3835, 6251, 1012, 102]

we use this in bert rune. You can find more details in our article.

14. utf8_decode#

A proc block which can convert u8 bytes to utf8.

image: runicos/base
version: 1
pipeline:
number_list:
capability: RAW
outputs:
- type: u8
dimensions: [50]
utf8_decode:
proc-block: "hotg-ai/proc-blocks#utf8_decode"
inputs:
- number_list
outputs:
- type: utf8
dimensions: [50]
serial:
out: SERIAL
inputs:
- utf8_decode

After building a rune using above Runefile run rune run test.rune --raw input.txt, where input .txt contains one two three four five. You will get this as output.

{"type_name":"utf8","channel":1,"elements":["one two three four five"],"dimensions":[1]}

The capability convert the input data to u8 bytes so we can use this proc-block to convert them back to utf8