阿里云存储服务OSS API（python）

Posted on by

本文将从阿里云存储中基本概念，python API 和几个例子来简述阿里云python API 的使用，使新手能够简单上手阿里云的云存储。

阿里云存储中的基本概念

对象 Object

阿里云存储中，任何一个存储的文件都是一个对象，包含了key、data 和user meta，分别对应着对象的名称、数据和对象描述。对象名使用utf-8 编码，且长度在1-1023 之间。操作对象的操作有：get、range query、delete、list。

对象组 Object Group

多个对象组成的松散集合。可以像操作对象一样操作对象组。

桶 Bucket

桶就是用来装对象数据的，在整个阿里云存储服务（OSS）中具有全局唯一性，相当于是阿里云给每个用户的二级域名，每个用户最多可以创建10 个桶。所有的对象都是放在桶中的，至于能放多少没有限制。桶的命名包括小写字母、数字、下划线和短横线，必须以小写字母或者数字开头。通过浏览器设置存储空间长度只能在6-16 个字符内，而通过python 的API 可以设置在3-255 个字符内。

Access ID 和Access key

在OSS 注册时，用户会给用户分配Access ID 和Access key 分别用来标识用户和认证用户用。Access key 是作为私钥发送给用户，每次用户为认证自己，将一段随机数字加上用私钥加密随机数字的结果发送给服务器端，服务器用用户的公钥解密对应的加密结果，如果和随机数字一致则用户得到验证。服务器端如果公钥泄密问题不大，但如果用户的私钥泄露出去了则就要更换公私密钥对了，生成公私密钥对的大素数对是可以不变的，虽然这也有一定风险。【如果不明白上面一段说了什么可以忽略】

总结一下：

  • Bucket 、Object 和Object Group 都支持的操作：创建（上传）、查看、list、删除
  • 可修改Bucket 访问权限（三种：私有、可读不可写、可读可写）
  • 访问时支持If-Modified-Since 和If-Match 等HTTP 参数

OSS API SDK

包含的py 文件

这部分将根据阿里云存储服务提供的python API 看看支持哪些操作，下载python SDK 看到压缩包里包含这么几个文件：oss_api.py  oss_cmd.py  oss_fs.py  oss_sample.py  oss_util.py  oss_xml_handler.py ，作用分别如下：

oss_api.py    用户直接使用该文件的类和方法操作Object、Bucket 等

oss_util.py    通用方法类，包含不被直接调用，但比较常用的类和方法

oss_xml_handler.py   阿里云存储的错误信息都是xml 格式的，用来处理xml 的

oss_cmd.py   用户可以以命令行的形式操作Object、Bucket 【optional】

oss_fs.py       利用OSS 搭建的一个简单的云存储文件系统 【optional】

oss_sample.py   简单的一些实用API 的例子【optional】

需要说明的是文件里面使用的类和方法有的比较古老了，编译的时候可能出现错误，比如oss_util.py 中md5 已经合并到hashlib 类了，md5.new() 方法也没有了，在编译oss_sample.py 的时候就会发现这样的错误。

包含的主要方法

看看oss_api.py 提供了哪些类和方法：整个文件就一个OssAPI 类，在使用该API 时首先应该通过

oss = OssAPI(HOST, ACCESS_ID, SECRET_ACCESS_KEY)

的方法生成一个该类的实例。该类对外公开的API 有以下方法：

sign_url_auth_with_expire_time(self, method, url, headers = {}, resource=”/”, timeout = 60)

通过输入的方法、url、body 和headers 创建认证，返回签名链接，method 方法有PUT, GET, DELETE, HEAD。这类似于生成一个公开的限时签名链接地址。

bucket_operation(self, method, bucket, headers={}, params={})

发送桶bucket 操作请求，method 方法有PUT, GET, DELETE, HEAD 。

object_operation(self, method, bucket, object, headers = {}, data=””)

发送对象object 操作请求，method 方法有PUT, GET, DELETE, HEAD 。

get_service(self)

list 所有buckets，等价调用list_all_my_buckets(self) 。

get_bucket_acl(self, bucket)

获取bucket 访问控制权限，有public-read-write，public-read 和private 三类。

get_bucket(self, bucket, prefix=”, marker=”, delimiter=”, maxkeys=”, headers = {})

list bucket 中所有对象，等价调用list_bucket() ，因此参数、结果相同。

create_bucket(self, bucket, acl=”, headers = {})

创建bucket，实际上是调用的put_bucket() 。

delete_bucket(self, bucket)

删除bucket 。

put_object_with_data(self, bucket, object, input_content, content_type=DefaultContentType, headers = {})

向对象中写入数据，等价调用put_object_from_string() 。

put_object_from_file(self, bucket, object, filename, content_type=DefaultContentType, headers = {})

从文件读取写入到对象中，实际上就是上传该文件，对应的参数是文件绝对路径。

put_object_from_fp(self, bucket, object, fp, content_type=DefaultContentType, headers = {})

从文件指针写入数据到对象，和上面方法不同是的参数为文件指针。

get_object(self, bucket, object, headers = {})

获取一个对象，即是以GET 方式调用object_operation() 方法。

get_object_to_file(self, bucket, object, filename, headers = {})

获取一个对象并写入到文件，即下载文件，DELETE 方式调用object_operation() 。

head_object(self, bucket, object, headers = {})

获取对象的头信息，而不用下载整个对象内容。

post_object_group(self, bucket, object, object_group_msg_xml, headers = {}, params = {})

上传对象组，将所有在object_group_msg_xml 中的对象合并为一个对象上传。

get_object_group_index(self, bucket, object, headers = {})

获取对象组索引，通过GET 方式的object_operation() 方法获得。

put_object_from_file_given_pos(self, bucket, object, filename, offset, partsize, content_type=DefaultContentType, headers = {})

从特定文件filename 的指定位置offset 读取指定大小partsize 到对象，并上传到桶bucket 中。

upload_large_file(self, bucket, object, filename, thread_num = 10, max_part_num = 1000)

上传一个大文件，分为1000 或者很多份，分别上传到桶bucket 中，最终合并多份内容为一个大的对象。

响应信息

conn 是httplib.HTTPConnection(self.host) 返回的一个连接，通过制定方法和报头可以操作self.host 网站，通过res = conn.getresponse() 获得响应结果。res 具有以下属性和方法：

                                      res.version

                                      res.reason

                                      res.status

                                      res.msg

                                      res.getheaders()

和服务器操作的方法会返回一个响应值，即是HTTP 协议回应的响应状态代码（res.status），三位数表示，第一个数代表响应类别：

                                     1XX：指示信息，表示请求已接收，继续处理

                                     2XX：成功，表示请求已成功接收，理解和接受

                                     3XX：重定向，要求请求必须进一步操作

                                     4XX：客户端错误，请求有语法错误或者请求无法实现

                                     5XX：服务器端错误，服务器未能实现合法的请求

这是一个大致的响应信息，如果需要更详细的了解，参考错误响应文档。

几个例子

在自带的oss_sample.py 就包含了一些主要操作的例子，可以参考借鉴。下面给出几个常见操作的范例：

list buckets  罗列所有存储空间

   1: from oss_api import *

   2: from oss_xml_handler import *

   3:  

   4: HOST="storage.aliyun.com"

   5: ACCESS_ID = ""

   6: SECRET_ACCESS_KEY = ""

   7:  

   8: oss = OssAPI(HOST, ACCESS_ID, SECRET_ACCESS_KEY)

   9:  

  10: res = oss.list_all_my_buckets()

  11: if 2 == (res.status / 100):

  12:     http_body = res.read()

  13:     bucket_list = GetServiceXml(http_body)

  14:     for bucket in bucket_list.list():

  15:         print bucket

  16: else:

  17:     print "ERROR"

 

list objects 罗列存储空间中所有对象

   1: from oss_api import *

   2: from oss_xml_handler import *

   3:  

   4: HOST="storage.aliyun.com"

   5: ACCESS_ID = ""

   6: SECRET_ACCESS_KEY = ""

   7:  

   8: oss = OssAPI(HOST, ACCESS_ID, SECRET_ACCESS_KEY)

   9:  

  10: res = oss.list_bucket(bucket_name)

  11: if 2 == (res.status / 100):

  12:     data = res.read()

  13:     h = GetBucketXml(data)

  14:     (file_list, common_list) = h.list()

  15:     for each in file_list:

  16:         print each

  17: 

 

create a bucket  创建一个存储空间

   1: from oss_api import *

   2: from oss_xml_handler import *

   3: import sys

   4:  

   5: HOST="storage.aliyun.com"

   6: ACCESS_ID = ""

   7: SECRET_ACCESS_KEY = ""

   8:  

   9: oss = OssAPI(HOST, ACCESS_ID, SECRET_ACCESS_KEY)

  10: new_bucket = sys.argv[1]

  11: res = oss.put_bucket(new_bucket)

  12: if 2 == (res.status / 100) :

  13:     print "Succeed"

  14: else: 

  15:     print "Fail\n%s" % res.read()

 

修改指定存储空间访问控制权限

因为创建存储空间默认为private ，需要自己设置权限

   1: from oss_api import *

   2: from oss_xml_handler import *

   3: import sys

   4:  

   5: HOST="storage.aliyun.com"

   6: ACCESS_ID = ""

   7: SECRET_ACCESS_KEY = ""

   8: bucket_name = sys.argv[1]

   9:  

  10: oss = OssAPI(HOST, ACCESS_ID, SECRET_ACCESS_KEY)

  11: res = oss.put_bucket(bucket_name, "public-read")

  12:  

  13: if 2 == (res.status / 100) :

  14:     print "Succeed"

  15: else:

  16:     print "Fail\n%s" % res.read()

 

上传一个文件到存储空间

   1: from oss_api import *

   2: from oss_xml_handler import *

   3:  

   4: HOST="storage.aliyun.com"

   5: ACCESS_ID = ""

   6: SECRET_ACCESS_KEY = ""

   7: bucket_name = sys.argv[1]

   8: file_name = sys.argv[2]

   9:  

  10: oss = OssAPI(HOST, ACCESS_ID, SECRET_ACCESS_KEY)

  11:  

  12: res = oss.put_object_from_file(bucket_name, "1.jpg", file_name, "image/jpg")

  13: if 2 == (res.status / 100):

  14:     print "Succeed"

  15: else: 

  16:     print "Fail\n%s" % res.read()

 

从存储空间下载指定文件

   1: from oss_api import *

   2: from oss_xml_handler import *

   3:  

   4: HOST="storage.aliyun.com"

   5: ACCESS_ID = ""

   6: SECRET_ACCESS_KEY = ""

   7:  

   8: oss = OssAPI(HOST, ACCESS_ID, SECRET_ACCESS_KEY)

   9: res = oss.get_object_to_file("hust", "1.jpg", "./new.JPEG")

  10: if 2 == (res.status / 100):

  11:     print "Succeed"

  12: else:

  13:     print "Fail\n%s" % res.read()

 

从存储空间删除指定文件

API 没有直接提供像delete_bucket() 样的delete_object() 的操作，但是用object_operation() 也是一样的，实际上delete_bucket() 也只是调用DELETE 方式的bucket_operation() 方法。

   1: from oss_api import *

   2: from oss_xml_handler import *

   3:  

   4: HOST="storage.aliyun.com"

   5: ACCESS_ID = ""

   6: SECRET_ACCESS_KEY = ""

   7:  

   8: oss = OssAPI(HOST, ACCESS_ID, SECRET_ACCESS_KEY)

   9: res = oss.object_operation("DELETE","hust","1.jpg")

  10: if 2 == (res.status/100):

  11:         print "Succeed!"

  12: else:

  13:         print res.read()

 

合并对象（object）为对象组（object group）

   1: from oss_api import *

   2: from oss_xml_handler import *

   3:  

   4: HOST="storage.aliyun.com"

   5: ACCESS_ID = ""

   6: SECRET_ACCESS_KEY = ""

   7:  

   8: obs_msg_list = [[0,"800.png","43B0177C78BF904970F7B1C9005001F3"],[1,"new.JPEG","5C6E71FDF26A82FBD42BC5D31787FE20"]]

   9:  

  10: xml_string = r'<;CreateFileGroup>'

  11: for part in obs_msg_list:

  12:     if isinstance(part[1], unicode):

  13:         file_path = part[1].encode('utf-8')

  14:     else:

  15:         file_path = part[1]

  16:         xml_string += r'<;Part>'

  17:         xml_string += r'<;PartNumber>' + str(part[0]) + r'</PartNumber>'

  18:         xml_string += r'<;PartName>' + str(file_path) + r'</PartName>'

  19:         xml_string += r'<;ETag>"' + str(part[2]).upper() + r'"</ETag>'

  20:         xml_string += r'<;/Part>'

  21: xml_string += r'<;/CreateFileGroup>'

  22:  

  23: oss = OssAPI(HOST, ACCESS_ID, SECRET_ACCESS_KEY)

  24: res = oss.post_object_group('hust','group.msh',xml_string)

  25: if 2 == (res.status / 100):

  26:     print "Successed!"

  27: else:

  28:     print res.read()

开始我试图自己写xml <CreateFileGroup>………</CreateFileGroup>，老是返回 InvalidXMLFormat ，估计还是编码的问题。

来个复杂点的

下面对上面的操作进行了封装，能够提供一个简单的命令行操作，写的有点乱。

   1: #coding=utf8

   2: from oss_api import *

   3: from oss_xml_handler import *

   4: import os

   5: import sys

   6:  

   7: HOST="storage.aliyun.com"

   8: ACCESS_ID = ""

   9: SECRET_ACCESS_KEY = ""

  10:  

  11: class OssFS:

  12:     def __init__(self, oHost, oId="", oKey=""):

  13:         self.oHost = oHost

  14:         self.oId = oId

  15:         self.oKey = oKey

  16:         self.oss = OssAPI(oHost,oId,oKey)

  17:         self.buckets = []

  18:         res = self.oss.list_all_my_buckets()

  19:         http_body = res.read()

  20:         if 2 == (res.status/100):

  21:             bucket_list = GetServiceXml(http_body)

  22:             for bucket in bucket_list.list():

  23:                 (h1,h2) = bucket

  24:                 self.buckets.append(str(h1))

  25:         else:

  26:                 print http_body

  27:  

  28:     def show_buckets(self):

  29:         '''Show all buckets'''

  30:         for each in self.buckets:

  31:             print each

  32:  

  33:     def sizetoKMGT(self,snum):

  34:         '''convert size number to KB/MB/GB/TB'''

  35:         if 0 == int(snum):

  36:             return "obj grp"

  37:         lsize = ['Bytes', 'KB', 'MB', 'GB', 'TB']

  38:         i = 0

  39:         snum = int(snum) * 1.0

  40:         while snum >; 1024:

  41:             snum = snum / 1024.0

  42:             i += 1;

  43:         return ('%.2f' + " " +lsize[i])%(snum)

  44:  

  45:  

  46:     def show_objects(self, bucket = '', path = ''):

  47:         '''Show all objects in the path from the bucket'''

  48:         res = self.oss.list_bucket(bucket)

  49:         http_body = res.read()

  50:         if 2 == (res.status / 100):

  51:             h = GetBucketXml(http_body)

  52:             (file_list ,common_list) = h.list()

  53:             for each in file_list:

  54:                 (fname, ctime, etag, fsize, owner, owner2, fstyle) = each

  55:                 print "%10s\t %s\t %s" % (str(fname),self.sizetoKMGT(fsize),etag)

  56:         else:

  57:             print http_body

  58:  

  59:     def file_upload(self, bucket='' ,path='' ,file_name=''):

  60:         '''Upload a file from local PC to the OSS'''

  61:         res = self.oss.put_object_from_file(bucket, file_name, file_name)

  62:         if 2 == (res.status/100): 

  63:             print "Success to upload the file: " + file_name

  64:             return True

  65:         else:

  66:             print res.read()

  67:     

  68:     def is_file(self, bucket, file_name):

  69:         '''Wether the file_name is in the bucket or not'''

  70:         res = self.oss.list_bucket(bucket)

  71:         http_body = res.read()

  72:         if 2 ==(res.status / 100):

  73:             h = GetBucketXml(http_body)

  74:             (file_list, common_list) = h.list()

  75:             for each in file_list:

  76:                 fname = str(each[0])

  77:                 if fname == file_name:

  78:                     return True

  79:             return False

  80:  

  81:     def file_download(self, bucket, file_name):

  82:         '''Download the file(file_name) from the bucket'''

  83:         res = self.get_object_to_file(bucket, file_name, file_name)

  84:         if 2 == (res.status / 100):

  85:             print "Get file: "+file_name

  86:         else:

  87:             print res.read()

  88:     

  89:     def file_delete(self, bucket, file_name):

  90:         '''Delete the file(file_name) from the bucket'''

  91:         res = self.oss.object_operation("DELETE", bucket, file_name)

  92:         if 2 == (res.status/100):

  93:             print "Success to delete the file: " + file_name

  94:             return True

  95:         else:

  96:             print res.read()

  97:             return False

  98:  

  99: if __name__=="__main__":

 100:     isRoot = True;

 101:     #isRoot wether is root floder

 102:     uBucket = ''

 103:     uPath = ''

 104:     uFs = OssFS(HOST,ACCESS_ID,SECRET_ACCESS_KEY)

 105:  

 106:     print "==============================================="

 107:     while True:

 108:         strCmd = raw_input("Input your command:")

 109:         cmd = strCmd.split(' ')

 110:  

 111:         if "ls" == cmd[0]:

 112:             if 1 == len(cmd):

 113:                 if True == isRoot:

 114:                     uFs.show_buckets()

 115:                     continue

 116:                 else:

 117:                     uFs.show_objects(uBucket,uPath)

 118:                     continue

 119:             elif 2 == len(cmd):

 120:                 if "local" == cmd[1]:

 121:                     for each in os.listdir('.'):

 122:                         print each

 123:                     continue

 124:                 else:

 125:                     print ">>>ERROR: BAD option! (Try \"ls local\")"

 126:                     continue

 127:  

 128:         elif "cd" == cmd[0]:

 129:             if len(cmd) != 2:

 130:                 print ">>>ERROR: BAD option! (Bucket name or file name need)"

 131:                 continue

 132:             else:

 133:                 if '.' == cmd[1]:

 134:                     continue

 135:                 elif '..' == cmd[1]:

 136:                     if True == isRoot:

 137:                         print "Root already"

 138:                     else:

 139:                         isRoot = True

 140:                         continue

 141:                 else:

 142:                     if False == isRoot:

 143:                         print ">>>ERROR BAD bucket, you are in bucket already"

 144:                         continue

 145:                     uBucket = cmd[1]

 146:                     if uBucket not in uFs.buckets:

 147:                         print ">>>ERROR BAD bucket"+uBucket+" is not in your buckets"

 148:                         continue

 149:                     isRoot = False

 150:                     continue

 151:  

 152:         elif "put" == cmd[0]:

 153:             if True == isRoot:

 154:                 print ">>>ERROR: BAD PATH (you must cd a bucket first )"

 155:                 continue

 156:             if len(cmd) != 2:

 157:                 print ">>>ERROR: BAD option! (put a file one time)"

 158:                 continue

 159:             else:

 160:                 file_name = cmd[1]

 161:                 if os.path.isfile(file_name):                    #file exist

 162:                     uFs.file_upload(uBucket,uPath,file_name)

 163:                     continue

 164:                 else:

 165:                     print ">>>ERROR: BAD file name (file not exist!)"

 166:                     continue

 167:  

 168:         elif "get" == cmd[0]:

 169:             if len(cmd) !=2:

 170:                 print ">>>ERROR: BAD option"

 171:                 continue

 172:             if True == isRoot:

 173:                 print ">>>ERROR: BAD get, access a bucket first"

 174:                 continue

 175:             file_name = cmd[1]

 176:             if False == uFs.is_file(uBucket, file_name):

 177:                 print file_name + " is not in your bucket" + uBucket

 178:                 continue

 179:             uFs.file_download(uBacket,file_name)    #didnot consider the file has already exist

 180:             continue

 181:         

 182:         elif "delete" == cmd[0]:

 183:             if True == isRoot:

 184:                 print ">>>BAD delete, I will not let you delete your bucket！"

 185:                 continue

 186:             if len(cmd) != 2:

 187:                 print ">>>BAD options"

 188:                 continue

 189:             file_name = cmd[1]

 190:             if False == uFs.is_file(uBucket, file_name):

 191:                 print file_name + " is not in your bucket" + uBucket

 192:                 continue

 193:             uFs.file_delete(uBucket, file_name)

 194:             continue

 195:             

 196:         elif "help" == cmd[0]:

 197:             pass

 198:             continue

 199:  

 200:         elif "quit" == cmd[0]:

 201:             break

 202:  

 203:         else:

 204:             print "Unable to recognize your command!"

 205:             continue

 206:     print "==============================================="

《阿里云存储服务OSS API（python）》上有2条评论

    • 这是比较老的API了，建议你去官网下载最新的API 和文档，比我这应该详细很多。

      回复 ↓

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注

此站点使用Akismet来减少垃圾评论。了解我们如何处理您的评论数据。