当前位置：网站首页>Upload files to remote devices through pyro4 command parameters

Upload files to remote devices through pyro4 command parameters

2022-07-19 15:27:00 【longyu_ wlz】

demand

Rules need to be distributed to local devices through the cloud , Because file channels are not supported , You need to issue rules to local devices through command parameters .

There are the following constraints ：

  1. Between the cloud and local devices python Pyro4 Framework to make remote calls
  2. The rule is ebpf Instruction code , Size in kb Level
  3. python Version is python2

How to achieve ？

After a thought and Exchange , Confirm the implementation scheme as follows ：

In the cloud ebpf Script use gzip Compress , And then use base64 Encoded as ascii The code is passed to the script of the local device through parameters , The script decodes the parameters , And then use gzip Decompress and store it in the specified location on the device .

At the same time, in order to ensure safety , increase md5sum check , Source of file md5sum Values are also passed by command , The script of the local device will verify the source file md5sum, If the verification fails, delete the file .

Example demo

#!/usr/bin/python

import sys
import os
from hashlib import md5
import base64
import subprocess
from subprocess import check_call

def generate_file_md5value(fpath):
    m = md5()
    a_file = open(fpath, 'rb')
    m.update(a_file.read())
    a_file.close()
    return m.hexdigest()

def ebpf_base64_code_save(ebpf_code, file_md5sum):
    if len(file_md5sum) == 0:
        return 'must give ebpfcode md5sum value'

    ebpfcode = base64.b64decode(ebpf_code)
    filepath = "/tmp/ebpf.o.gz"
    f = open(filepath, "wb")
    f.write(ebpfcode)
    f.close()
    try:
        check_call('gzip -d -f /tmp/ebpf.o.gz', shell=True)
    except subprocess.CalledProcessError as exc:
        check_call('rm -rf /tmp/ebpf.o.gz')
        return exc.output

    md5sum = generate_file_md5value("/tmp/ebpf.o")

    if md5sum != file_md5sum:
        os.system('rm -rf /tmp/ebpf.o')
        return 'md5sum failed\\n'

    return 'succeed'

def upload_ebpfcode_test(filename):
    md5sum = generate_file_md5value(filename)

    command = 'gzip ' + filename
    check_call(command, shell=True)

    gzip_filename = filename + '.gz'

    f = open(gzip_filename, "rb")
    component = f.read();
    f.close();

    ebpf_code = base64.b64encode(component)

    command = 'gzip -d -f ' + filename
    check_call(command, shell=True)

    return ebpf_base64_code_save(ebpf_code, md5sum)

def main():
    if (len(sys.argv) > 1):
        print(upload_ebpfcode_test(sys.argv[1]))
    else:
        print("Please input an filename")

if __name__ == "__main__":
    main()

Example demo Main process

upload_ebpfcode_test Calculate the... Of the source file at the beginning of the function md5sum value , Then use the specified file gzip After compression base64 code , Finally using base64 The content of the code is similar to md5sum Value is called as a parameter ebpf_base64_code_save function .

ebpf_base64_code_save Function first decodes ebpf_code The rule file represented by the parameter , And then use gzip Unzip the file , Verify the file after decompression md5sum, If the verification fails, delete the file .

Example demo test

  1. Generate a file 

    [[email protected]:15:03:34] tmp $ dd if=/dev/random  of=./data bs=1M count=12
12+0 records in
12+0 records out
12582912 bytes (13 MB, 12 MiB) copied, 0.0874574 s, 144 MB/s

  2. perform demo Script 

    [[email protected]:15:01:57] tmp $ python ./demo.py data
succeed

  3. View results 

    [[email protected]:15:01:59] tmp $ md5sum ./data /tmp/ebpf.o 
b06ea768af2f76b468b7e3f0fadcb795  ./data
b06ea768af2f76b468b7e3f0fadcb795  /tmp/ebpf.o

md5sum Same value , The test passed .

The local structures, Pyro4 Environmental simulation

I'm trying to debian11 Use in python Pyro4 To build an environment close to the actual implementation process , Here are some key points .

The sample code is written with reference to the following official documents ：

Intro and Example - Pyro 4.82 documentation

python install Pyro4

Execute the following command to install ：

[[email protected]:11:34:20] tmp $ sudo pip instlal Pyro4

Pyro4 The test script

Server side server.py Source code :

# saved as greeting-server.py

import sys
import os
from hashlib import md5
import base64
import subprocess
from subprocess import check_call
import Pyro4
import serpent

def generate_file_md5value(fpath):
    m = md5()
    a_file = open(fpath, 'rb')
    m.update(a_file.read())
    a_file.close()
    return m.hexdigest()

def ebpf_base64_code_save(ebpf_code, file_md5sum):
    if len(file_md5sum) == 0:
        return 'must give ebpfcode md5sum value'

    ebpfcode = base64.b64decode(ebpf_code)
    filepath = "/tmp/ebpf.o.gz"
    f = open(filepath, "wb")
    f.write(ebpfcode)
    f.close()
    try:
        check_call('gzip -d -f /tmp/ebpf.o.gz', shell=True)
    except subprocess.CalledProcessError as exc:
        check_call('rm -rf /tmp/ebpf.o.gz')
        return exc.output

    md5sum = generate_file_md5value("/tmp/ebpf.o")

    if md5sum != file_md5sum:
        os.system('rm -rf /tmp/ebpf.o')
        return 'md5sum failed\n'

    return 'succeed'

@Pyro4.expose
class GreetingMaker(object):
    def save_ebpfcode(self, argument):
        ebpf_code = argument[0]
        ebpf_code = ebpf_code['data'].strip()
        md5sum = argument[1]
        return ebpf_base64_code_save(ebpf_code, md5sum)
    def hello(self):
        return 'hello world\n'

daemon = Pyro4.Daemon()                # make a Pyro daemon
uri = daemon.register(GreetingMaker)   # register the greeting maker as a Pyro object

print("Ready. Object uri =", uri)      # print the uri so we can use it in the client later
daemon.requestLoop()                   # start the event loop of the server to wait for calls

client client.py Source code ：

# saved as greeting-client.py
import sys
import os
from hashlib import md5
import base64
import subprocess
from subprocess import check_call
import Pyro4

def generate_file_md5value(fpath):
    m = md5()
    a_file = open(fpath, 'rb')
    m.update(a_file.read())
    a_file.close()
    return m.hexdigest()

uri = input("What is the Pyro uri of the greeting object? ").strip()
if (len(sys.argv[1]) > 1):
    filename = sys.argv[1]
else:
    print('Please input filename\n')
    os.exit(-1)

greeting_maker = Pyro4.Proxy(uri)         # get a Pyro proxy to the greeting object

md5sum = generate_file_md5value(filename)
command = 'gzip ' + filename
check_call(command, shell=True) 

gzip_filename = filename + '.gz'
f = open(gzip_filename, "rb")
component = f.read();
f.close();

command = 'gzip -d -f ' + filename
check_call(command, shell=True)

argument = [component, md5sum]
print(greeting_maker.save_ebpfcode(argument))

Testing process

  1. Running server 

    [[email protected]:13:21:03] tmp $ python3 ./server.py 
Ready. Object uri = PYRO:[email protected]:38207

    Printed PYRO:[email protected]:38207 Contains connection information , It will be used when the client connects .

  2. Running client 

    [[email protected]:13:22:51] tmp $ python3 ./client.py data
What is the Pyro uri of the greeting object? PYRO:[email protected]:38207
succeed

    The client needs to input the information printed by the server uri Information to establish a connection .

  3. file md5sum check 

    [[email protected]:13:23:03] tmp $ md5sum ./data /tmp/ebpf.o 
c4fa672da73f38f9441d2c9638848269  ./data
c4fa672da73f38f9441d2c9638848269  /tmp/ebpf.o

Some problems found in the test are recorded

  1. python Version of the problem

    Use pip Install well Pyro4 module after , I try to use python Come on import This module tests , It turns out that this module does not exist , Try to find out in my system pip The order is python3 The order of , And the default. python The link points to python2, cause Pyro4 Module cannot be imported . I didn't notice what I used here python The version is inconsistent with the version of the real environment .

  2. Pyro4 Function parameter transfer problem

    At the beginning, I was in server.py Of save_ebpfcode Three parameters are defined in the function , It is not supported when using , You need to splice multiple parameters into one list To pass on .

  3. Why? client.py Compressed files are not base64 code ？

    At first I used demo.py The code in base64 code , Then pass it to the remote call parameters , The test found that the decompressed file always reported that the file was not gzip Compressed format .

    After an analysis, it is found that python3 in base64.b64encode(component) The data type returned is binary Format , This format is in Pyro4 There will be the following problems ：

    Insert picture description here

    To put it simply , When the parameter type is binary When ,Pyro4 Parameters will be base64 Code and convert to {'data': 'aXJtZW4gZGUgam9uZw==', 'encoding': 'base64'} This format , On the surface, it looks like data key Of value It's the parameters that are passed , In fact, this parameter is executed again base64 code .

    So in client.py Directly pass the compressed file contents as parameters , At the same time server.py The parsing code in is modified as follows ：

        def save_ebpfcode(self, argument):
        ebpf_code = argument[0]
        ebpf_code = ebpf_code['data'].strip()
        md5sum = argument[1]
        return ebpf_base64_code_save(ebpf_code, md5sum)

python2 Use Pyro4 To test

debian11 Installation on python2 pip And python2 Pyro4 module

I refer to https://blog.emacsos.com/pip2-in-debian-11-bullseye.html The content of this link is installed , Use wget download get-pip.py Found that the speed is very slow , Try to configure socks5 The agent found wget Does not support , Try to use tsocks When you command, you think you can use the browser with the agent configured to download directly , It's really fast .

The installation process is as follows ：

[[email protected]:14:25:51] Downloads $ python2 ./get-pip.py 
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Defaulting to user installation because normal site-packages is not writeable
Collecting pip<21.0
  Downloading pip-20.3.4-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 11 kB/s 
Collecting setuptools<45
  Using cached setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
Collecting wheel
  Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: pip, setuptools, wheel
  WARNING: The scripts pip, pip2 and pip2.7 are installed in '/home/longyu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts easy_install and easy_install-2.7 are installed in '/home/longyu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script wheel is installed in '/home/longyu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-20.3.4 setuptools-44.1.1 wheel-0.37.1

Execute the following command to install Pyro4 module：

[[email protected]:14:28:56] Downloads $ python2 -m pip install Pyro4
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Defaulting to user installation because normal site-packages is not writeable
Collecting Pyro4
  Using cached Pyro4-4.82-py2.py3-none-any.whl (89 kB)
Collecting serpent<1.30,>=1.27; python_version < "3.2"
  Downloading serpent-1.28-py2.py3-none-any.whl (11 kB)
Collecting selectors34; python_version < "3.4"
  Downloading selectors34-1.2-py2.py3-none-any.whl (8.2 kB)
Collecting six
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: serpent, six, selectors34, Pyro4
  WARNING: The scripts pyro4-check-config, pyro4-flameserver, pyro4-httpgateway, pyro4-ns, pyro4-nsc and pyro4-test-echoserver are installed in '/home/longyu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Pyro4-4.82 selectors34-1.2 serpent-1.28 six-1.16.0

Test code

Server side server.py Source code :

# saved as greeting-server.py

import sys
import os
from hashlib import md5
import base64
import subprocess
from subprocess import check_call
import Pyro4
import serpent

def generate_file_md5value(fpath):
    m = md5()
    a_file = open(fpath, 'rb')
    m.update(a_file.read())
    a_file.close()
    return m.hexdigest()

def ebpf_base64_code_save(ebpf_code, file_md5sum):
    if len(file_md5sum) == 0:
        return 'must give ebpfcode md5sum value'

    ebpfcode = base64.b64decode(ebpf_code)
    filepath = "/tmp/ebpf.o.gz"
    f = open(filepath, "wb")
    f.write(ebpfcode)
    f.close()
    try:
        check_call('gzip -d -f /tmp/ebpf.o.gz', shell=True)
    except subprocess.CalledProcessError as exc:
        check_call('rm -rf /tmp/ebpf.o.gz')
        return exc.output

    md5sum = generate_file_md5value("/tmp/ebpf.o")

    if md5sum != file_md5sum:
        os.system('rm -rf /tmp/ebpf.o')
        return 'md5sum failed\n'

    return 'succeed'

@Pyro4.expose
class GreetingMaker(object):
    def save_ebpfcode(self, argument):
        ebpf_code = argument[0]
        md5sum = argument[1]
        return ebpf_base64_code_save(ebpf_code, md5sum)
    def hello(self):
        return 'hello world\n'

daemon = Pyro4.Daemon()                # make a Pyro daemon
uri = daemon.register(GreetingMaker)   # register the greeting maker as a Pyro object

print("Ready. Object uri =", uri)      # print the uri so we can use it in the client later
daemon.requestLoop()                   # start the event loop of the server to wait for calls

client client.py Source code ：

# saved as greeting-client.py
import sys
import os
from hashlib import md5
import base64
import subprocess
from subprocess import check_call
import Pyro4

def generate_file_md5value(fpath):
    m = md5()
    a_file = open(fpath, 'rb')
    m.update(a_file.read())
    a_file.close()
    return m.hexdigest()

uri = input("What is the Pyro uri of the greeting object? ")
if (len(sys.argv[1]) > 1):
    filename = sys.argv[1]
else:
    print('Please input filename\n')
    os.exit(-1)

greeting_maker = Pyro4.Proxy(uri)         # get a Pyro proxy to the greeting object

md5sum = generate_file_md5value(filename)
command = 'gzip ' + filename
check_call(command, shell=True) 

gzip_filename = filename + '.gz'
f = open(gzip_filename, "rb")
component = f.read();
f.close();

ebpf_code = base64.b64encode(component)

command = 'gzip -d -f ' + filename
check_call(command, shell=True)

argument = [ebpf_code, md5sum]
print argument
print(greeting_maker.save_ebpfcode(argument))

Testing process

Running server side ：

[[email protected]:14:41:29] tmp $ python ./server.py 
('Ready. Object uri =', <Pyro4.core.URI at 0x7f0be2ed2810; PYRO:[email protected]:33951>)

Running client ：

[[email protected]:14:38:48] tmp $ python ./client.py ./test
What is the Pyro uri of the greeting object? 'PYRO:[email protected]:33951'
succeed

file md5sum contrast ：

[[email protected]:14:42:28] tmp $ md5sum ./test ./ebpf.o 
d8e8fca2dc0f896fd7cb4cb0031ba249  ./test
d8e8fca2dc0f896fd7cb4cb0031ba249  ./ebpf.o

Records of problems found during the test

  1. python2 in base64.b64encode The type of return value is u, Use Pyro4 When passing parameters, they will not be encoded again ,client.py The logic here and demo.py Agreement

  2. python2 Of input In output function , To enter special characters , You need to add single quotes

    take uri It can be entered normally if it is contained in a pair of single quotation marks , otherwise : And other special characters cannot be input ,python Script running error .

summary

When writing this article , Because of ignoring python The version of has taken many detours , In the end python3 And python2 The different test processes of are described .

Converting files into parameter passing is not quite in line with conventional cognition , But it can also be useful in specific scenes , The basic cognition that can do this is parameters and data , In network communication , Files are also data , There is no difference between the two .

Reference link

Intro and Example - Pyro 4.82 documentation

Why do I need ‘b’ to encode a string with Base64?

Tips & Tricks - Pyro 4.82 documentation

Install pip for Python2.7 in Debian 11 Bullseye

原网站

版权声明
本文为[longyu_ wlz]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/200/202207172332094569.html

边栏推荐

猜你喜欢

随机推荐