はじめに
Rook-Cephの設定や機能について、公式ドキュメントをベースにまとめています。今回はCephで利用できるストレージのうち、オブジェクトストレージを利用するために必要な情報を紹介します。
検証環境
- Rook version: ver 1.2
- Kubernetes version: 1.17.0
- Masterノード: 1台
- Workerノード: 3台
- Kubernetes環境: Azure上に仮想マシンを構築し、Kubeadmを用いてクラスターを作成
Object Storage
Rook-Cephでオブジェクトストレージを利用する場合、以下の流れで構築を行います。
CephClusterの作成CephObjectStoreの作成Storage Classの作成ObjectBucketClaimの作成- オブジェクトストレージの利用
1. CephClusterの作成
Rook-CephでObject Storageを利用する場合は、事前にCephClusterを作成する必要があります。作成するには公式ドキュメントのこちらの手順に従えばできます。Block Storageを利用するときにも同じ手順を踏むので、ここでは説明は省きます。
CephClusterを構築する際の利用するyamlファイルは以下になります。
apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: image: ceph/ceph:v14.2.5 allowUnsupported: true dataDirHostPath: /data skipUpgradeChecks: false continueUpgradeAfterChecksEvenIfNotHealthy: false mon: count: 1 allowMultiplePerNode: true dashboard: enabled: true ssl: true storage: useAllNodes: true useAllDevices: false config: databaseSizeMB: "1024" # this value can be removed for environments with normal sized disks (100 GB or larger) journalSizeMB: "1024" # this value can be removed for environments with normal sized disks (20 GB or larger) osdsPerDevice: "1" # this value can be overridden at the node or device level devices: - name: "sdd"
[root@vm0 ceph]# kubectl apply -f common.yaml [root@vm0 ceph]# kubectl apply -f operator.yaml [root@vm0 ceph]# kubectl apply -f cluster-test-clusterwide-device.yaml [root@vm0 ceph]# kubectl -n rook-ceph get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES csi-cephfsplugin-7gf76 3/3 Running 0 2m8s 10.0.1.4 vm3 <none> <none> csi-cephfsplugin-fsb25 3/3 Running 0 2m8s 10.0.1.5 vm1 <none> <none> csi-cephfsplugin-nfkls 3/3 Running 0 2m8s 10.0.1.7 vm2 <none> <none> csi-cephfsplugin-provisioner-8b9d48896-rkknx 4/4 Running 0 2m8s 10.244.2.132 vm2 <none> <none> csi-cephfsplugin-provisioner-8b9d48896-xw7x9 4/4 Running 0 2m8s 10.244.3.68 vm3 <none> <none> csi-rbdplugin-7tr7m 3/3 Running 0 2m8s 10.0.1.7 vm2 <none> <none> csi-rbdplugin-8b9bd 3/3 Running 0 2m8s 10.0.1.4 vm3 <none> <none> csi-rbdplugin-provisioner-6d465d6c6f-cs76m 5/5 Running 0 2m8s 10.244.1.64 vm1 <none> <none> csi-rbdplugin-provisioner-6d465d6c6f-hfwsp 5/5 Running 0 2m8s 10.244.3.69 vm3 <none> <none> csi-rbdplugin-qst72 3/3 Running 0 2m8s 10.0.1.5 vm1 <none> <none> rook-ceph-crashcollector-vm1-666498f4db-vqlps 1/1 Terminating 0 90s 10.244.1.66 vm1 <none> <none> rook-ceph-crashcollector-vm1-87d7886d4-mw2ss 1/1 Running 0 32s 10.244.1.69 vm1 <none> <none> rook-ceph-crashcollector-vm2-d9879bccc-wb28s 1/1 Running 0 100s 10.244.2.136 vm2 <none> <none> rook-ceph-crashcollector-vm3-5dddcf7994-42krj 1/1 Running 0 27s 10.244.3.72 vm3 <none> <none> rook-ceph-mgr-a-697799d58-q94s5 1/1 Running 0 90s 10.244.1.65 vm1 <none> <none> rook-ceph-mon-a-5f4dd48944-gnv84 1/1 Running 0 100s 10.244.2.135 vm2 <none> <none> rook-ceph-operator-678887c8d-4qbnw 1/1 Running 0 4m58s 10.244.1.62 vm1 <none> <none> rook-ceph-osd-0-7b84d6494d-4xn56 1/1 Running 0 32s 10.244.1.68 vm1 <none> <none> rook-ceph-osd-1-76fd5996df-6l2w9 1/1 Running 0 27s 10.244.3.71 vm3 <none> <none> rook-ceph-osd-2-85d969f9-dxhk9 1/1 Running 0 26s 10.244.2.138 vm2 <none> <none> rook-ceph-osd-prepare-vm1-q8n4d 0/1 Completed 0 69s 10.244.1.67 vm1 <none> <none> rook-ceph-osd-prepare-vm2-6ndxp 0/1 Completed 0 69s 10.244.2.137 vm2 <none> <none> rook-ceph-osd-prepare-vm3-fd4vq 0/1 Completed 0 69s 10.244.3.70 vm3 <none> <none> rook-discover-7kq7g 1/1 Running 0 4m54s 10.244.3.67 vm3 <none> <none> rook-discover-l7xrj 1/1 Running 0 4m54s 10.244.1.63 vm1 <none> <none> rook-discover-sj6np 1/1 Running 0 4m54s 10.244.2.131 vm2 <none> <none> [root@vm0 ceph]#
2. CephObjectStoreの作成
CephClusterを作成したら、次はCephObjectStoreリソースを作成します。
Object Store Settings
spec.metadataPool: メタデータプールに関する設定を指定。データ冗長化の方式はreplicatedのみ指定可能failureDomain: データ複製時のfailureDomainを指定。hostosdのほか、ノードに付与されたラベルも指定できる（利用可能なラベルはこちらを参照）replicated: データ複製数を指定
spec.dataPool: データプールに関する設定を指定。データ冗長化の方式はreplicatederasureCodedのいずれかを指定可能failureDomain: データ複製時のfailureDomainを指定replicated: データ複製数を指定erasureCoded: データ複製方式をErasure Codeにする場合の設定項目を指定dataChunks: 元のオブジェクトデータをいくつのチャンクに分割するかを指定codingChunks: 復旧用のチャンクをいくつ作成するかを指定
spec.preservePoolsOnDelete:trueの場合、CephObjectStoreが削除されてもプールは削除されない。デフォルトではfalse
※参考リンク：
Ceph Doc - Erasure Code Profile
Gateway Settings
spec.gateway: RADOS GWの設定内容を指定type: GWのタイプ。s3のみをサポートsslCertificateRef: SSLを利用する場合は証明書を指定port: RADOS GW PodがRADOS GW Serviceと通信する際に利用するポートsecurePort: SSL証明書を指定した場合のみ有効、RADOS GW PodがListenするポートを指定instances: Pod数を指定annotations: Podに付与するAnnotationを指定placement: Podの配置場所を操作する場合に設定nodeAffinity: KubernetesのNode Affinityを利用podAffinity: KubernetesのPod Affinityを利用podAntiAffinity: KubernetesのPod Anti-Affinityを利用tolerations: KubernetesのTaints/Tolerationsを利用
resources: Podのリソース制限を設定requests: リソース下限を指定。cpumemoryを指定可能limits: リソース上限を指定。cpumemoryを指定可能
priorityClassName: Priority Class Nameを指定
※参考リンク：
Kubernetes Doc - Assigning Pods to Nodes
Kubernetes Doc - Taints and Tolerations
サンプル例
以下のyamlファイルでは、メタデータ、データともにhostをfailureDomainとし、データの冗長性はreplicatedで3つの複製を作成します。またRADOS GW Podを1つ作成します。
apiVersion: ceph.rook.io/v1 kind: CephObjectStore metadata: name: my-store namespace: rook-ceph spec: # The pool spec used to create the metadata pools. Must use replication. metadataPool: failureDomain: host replicated: size: 3 # The pool spec used to create the data pool. Can use replication or erasure coding. dataPool: failureDomain: host replicated: size: 3 # Whether to preserve metadata and data pools on object store deletion preservePoolsOnDelete: false # The gateway service configuration gateway: # type of the gateway (s3) type: s3 # A reference to the secret in the rook namespace where the ssl certificate is stored sslCertificateRef: # The port that RGW pods will listen on (http) port: 80 # The port that RGW pods will listen on (https). An ssl certificate is required. securePort: # The number of pods in the rgw deployment instances: 1
上記ファイルをもとにCephObjectStoreを作成します。
# CephObjectStore デプロイ [root@vm0 ceph]# kubectl apply -f object.yaml cephobjectstore.ceph.rook.io/my-store created [root@vm0 ceph]# kubectl -n rook-ceph get cephobjectstore.ceph.rook.io NAME AGE my-store 50s [root@vm0 ceph]# kubectl -n rook-ceph get pod -l app=rook-ceph-rgw NAME READY STATUS RESTARTS AGE rook-ceph-rgw-my-store-a-798596d87f-cckp6 1/1 Running 0 28s
CephObjectStoreを作成すると、合わせてRADOS GW Pod、RADOS GW PodとCephObjectStoreとを接続するためのService、オブジェクトストアで利用するMIMEタイプ情報を格納したConfigMapが作成されます。
# Podの確認 [root@vm0 ceph]# kubectl get pods -n rook-ceph NAME READY STATUS RESTARTS AGE csi-cephfsplugin-7gf76 3/3 Running 0 4h2m csi-cephfsplugin-fsb25 3/3 Running 0 4h2m csi-cephfsplugin-nfkls 3/3 Running 0 4h2m csi-cephfsplugin-provisioner-8b9d48896-rkknx 4/4 Running 0 4h2m csi-cephfsplugin-provisioner-8b9d48896-xw7x9 4/4 Running 0 4h2m csi-rbdplugin-7tr7m 3/3 Running 0 4h2m csi-rbdplugin-8b9bd 3/3 Running 0 4h2m csi-rbdplugin-provisioner-6d465d6c6f-cs76m 5/5 Running 0 4h2m csi-rbdplugin-provisioner-6d465d6c6f-hfwsp 5/5 Running 0 4h2m csi-rbdplugin-qst72 3/3 Running 0 4h2m rook-ceph-crashcollector-vm1-87d7886d4-mw2ss 1/1 Running 0 4h rook-ceph-crashcollector-vm2-589b44d85f-7vvfk 1/1 Running 0 24s rook-ceph-crashcollector-vm2-d9879bccc-rc5nr 1/1 Terminating 0 54m rook-ceph-crashcollector-vm3-5dddcf7994-42krj 1/1 Running 0 4h rook-ceph-mgr-a-697799d58-q94s5 1/1 Running 0 4h1m rook-ceph-mon-a-5f4dd48944-gnv84 1/1 Running 0 4h1m rook-ceph-operator-678887c8d-4qbnw 1/1 Running 0 4h5m rook-ceph-osd-0-7b84d6494d-4xn56 1/1 Running 0 4h rook-ceph-osd-1-76fd5996df-6l2w9 1/1 Running 0 4h rook-ceph-osd-2-85d969f9-dxhk9 1/1 Running 0 4h rook-ceph-osd-prepare-vm1-4fdmb 0/1 Completed 0 3h1m rook-ceph-osd-prepare-vm2-jtqgj 0/1 Completed 0 3h1m rook-ceph-osd-prepare-vm3-s7r5l 0/1 Completed 0 3h1m rook-ceph-rgw-my-store-a-798596d87f-d4nvr 1/1 Running 0 24s ★ rook-ceph-tools-7f96779fb9-kjvpc 1/1 Running 0 3h36m rook-discover-7kq7g 1/1 Running 0 4h5m rook-discover-l7xrj 1/1 Running 0 4h5m rook-discover-sj6np 1/1 Running 0 4h5m # ConfigMapの確認 [root@vm0 ceph]# kubectl get cm -n rook-ceph NAME DATA AGE local-device-vm1 1 4h4m local-device-vm2 1 4h4m local-device-vm3 1 4h4m rook-ceph-csi-config 1 4h2m rook-ceph-mon-endpoints 4 4h1m rook-ceph-rgw-my-store-mime-types 1 8s ★ rook-config-override 1 4h1m # Serviceの確認 [root@vm0 ceph]# kubectl get svc -n rook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE csi-cephfsplugin-metrics ClusterIP 10.96.42.224 <none> 8080/TCP,8081/TCP 4h2m csi-rbdplugin-metrics ClusterIP 10.96.154.224 <none> 8080/TCP,8081/TCP 4h2m rook-ceph-mgr ClusterIP 10.96.152.170 <none> 9283/TCP 4h1m rook-ceph-mgr-dashboard ClusterIP 10.96.50.233 <none> 8443/TCP 4h1m rook-ceph-mon-a ClusterIP 10.96.219.41 <none> 6789/TCP,3300/TCP 4h1m rook-ceph-rgw-my-store ClusterIP 10.96.192.143 <none> 80/TCP 20s ★
Runtime Settings (MIME types)
CephObjectStoreを作成するとrook-ceph-rgw-<STORE-NAME>-mime-typesというConfigMapが作成されます。ここにはオブジェクトストアで利用するMIMEタイプの一覧がデータとして保存されており、デフォルトで約800種類が登録されています。
利用者が開発したなどの特別なMIMEタイプがある場合や、セキュリティを懸念して利用できるMIMEタイプを絞る場合は、kubectl editでConfigMapを直接編集することで、内容を変更することができます。
ConfigMap
rook-ceph-rgw-<STORE-NAME>-mime-typesの内容
[root@vm0 ceph]# kubectl describe cm -n rook-ceph rook-ceph-rgw-my-store-mime-types Name: rook-ceph-rgw-my-store-mime-types Namespace: rook-ceph Labels: <none> Annotations: <none> Data ==== mime.types: ---- application/activemessage application/andrew-inset ez application/annodex anx application/applefile application/atom+xml atom application/atomcat+xml atomcat application/atomicmail application/atomserv+xml atomsrv application/batch-SMTP application/bbolin lin application/beep+xml application/cals-1840 application/commonground application/cu-seeme cu application/cybercash application/davmount+xml davmount application/dca-rft application/dec-dx application/dicom dcm application/docbook+xml application/dsptype tsp application/dvcs application/ecmascript es application/edi-consent application/edi-x12 application/edifact application/eshop application/font-sfnt otf ttf application/font-tdpfr pfr application/font-woff woff application/futuresplash spl application/ghostview application/gzip gz application/hta hta application/http application/hyperstudio application/iges application/index application/index.cmd application/index.obj application/index.response application/index.vnd application/iotp application/ipp application/isup application/java-archive jar application/java-serialized-object ser application/java-vm class application/javascript js application/json json application/m3g m3g application/mac-binhex40 hqx application/mac-compactpro cpt application/macwriteii application/marc application/mathematica nb nbp application/mbox mbox application/ms-tnef application/msaccess mdb application/msword doc dot application/mxf mxf application/news-message-id application/news-transmission application/ocsp-request application/ocsp-response application/octet-stream bin deploy msu msp application/oda oda application/oebps-package+xml opf application/ogg ogx application/onenote one onetoc2 onetmp onepkg application/parityfec application/pdf pdf application/pgp-encrypted pgp application/pgp-keys key application/pgp-signature sig application/pics-rules prf application/pkcs10 application/pkcs7-mime application/pkcs7-signature application/pkix-cert application/pkix-crl application/pkixcmp application/postscript ps ai eps epsi epsf eps2 eps3 application/prs.alvestrand.titrax-sheet application/prs.cww application/prs.nprend application/qsig application/rar rar application/rdf+xml rdf application/remote-printing application/riscos application/rtf rtf application/sdp application/set-payment application/set-payment-initiation application/set-registration application/set-registration-initiation application/sgml application/sgml-open-catalog application/sieve application/sla stl application/slate application/smil+xml smi smil application/timestamp-query application/timestamp-reply application/vemmi application/whoispp-query application/whoispp-response application/wita application/x400-bp application/xhtml+xml xhtml xht application/xml xml xsd application/xml-dtd application/xml-external-parsed-entity application/xslt+xml xsl xslt application/xspf+xml xspf application/zip zip application/vnd.3M.Post-it-Notes application/vnd.accpac.simply.aso application/vnd.accpac.simply.imp application/vnd.acucobol application/vnd.aether.imp application/vnd.android.package-archive apk application/vnd.anser-web-certificate-issue-initiation application/vnd.anser-web-funds-transfer-initiation application/vnd.audiograph application/vnd.bmi application/vnd.businessobjects application/vnd.canon-cpdl application/vnd.canon-lips application/vnd.cinderella cdy application/vnd.claymore application/vnd.commerce-battelle application/vnd.commonspace application/vnd.comsocaller application/vnd.contact.cmsg application/vnd.cosmocaller application/vnd.ctc-posml application/vnd.cups-postscript application/vnd.cups-raster application/vnd.cups-raw application/vnd.cybank application/vnd.debian.binary-package deb ddeb udeb application/vnd.dna application/vnd.dpgraph application/vnd.dxr application/vnd.ecdis-update application/vnd.ecowin.chart application/vnd.ecowin.filerequest application/vnd.ecowin.fileupdate application/vnd.ecowin.series application/vnd.ecowin.seriesrequest application/vnd.ecowin.seriesupdate application/vnd.enliven application/vnd.epson.esf application/vnd.epson.msf application/vnd.epson.quickanime application/vnd.epson.salt application/vnd.epson.ssf application/vnd.ericsson.quickcall application/vnd.eudora.data application/vnd.fdf application/vnd.ffsns application/vnd.flographit application/vnd.font-fontforge-sfd sfd application/vnd.framemaker application/vnd.fsc.weblaunch application/vnd.fujitsu.oasys application/vnd.fujitsu.oasys2 application/vnd.fujitsu.oasys3 application/vnd.fujitsu.oasysgp application/vnd.fujitsu.oasysprs application/vnd.fujixerox.ddd application/vnd.fujixerox.docuworks application/vnd.fujixerox.docuworks.binder application/vnd.fut-misnet application/vnd.google-earth.kml+xml kml application/vnd.google-earth.kmz kmz application/vnd.grafeq application/vnd.groove-account application/vnd.groove-identity-message application/vnd.groove-injector application/vnd.groove-tool-message application/vnd.groove-tool-template application/vnd.groove-vcard application/vnd.hhe.lesson-player application/vnd.hp-HPGL application/vnd.hp-PCL application/vnd.hp-PCLXL application/vnd.hp-hpid application/vnd.hp-hps application/vnd.httphone application/vnd.hzn-3d-crossword application/vnd.ibm.MiniPay application/vnd.ibm.afplinedata application/vnd.ibm.modcap application/vnd.informix-visionary application/vnd.intercon.formnet application/vnd.intertrust.digibox application/vnd.intertrust.nncp application/vnd.intu.qbo application/vnd.intu.qfx application/vnd.irepository.package+xml application/vnd.is-xpr application/vnd.japannet-directory-service application/vnd.japannet-jpnstore-wakeup application/vnd.japannet-payment-wakeup application/vnd.japannet-registration application/vnd.japannet-registration-wakeup application/vnd.japannet-setstore-wakeup application/vnd.japannet-verification application/vnd.japannet-verification-wakeup application/vnd.koan application/vnd.lotus-1-2-3 application/vnd.lotus-approach application/vnd.lotus-freelance application/vnd.lotus-notes application/vnd.lotus-organizer application/vnd.lotus-screencam application/vnd.lotus-wordpro application/vnd.mcd application/vnd.mediastation.cdkey application/vnd.meridian-slingshot application/vnd.mif application/vnd.minisoft-hp3000-save application/vnd.mitsubishi.misty-guard.trustweb application/vnd.mobius.daf application/vnd.mobius.dis application/vnd.mobius.msl application/vnd.mobius.plc application/vnd.mobius.txf application/vnd.motorola.flexsuite application/vnd.motorola.flexsuite.adsi application/vnd.motorola.flexsuite.fis application/vnd.motorola.flexsuite.gotap application/vnd.motorola.flexsuite.kmr application/vnd.motorola.flexsuite.ttc application/vnd.motorola.flexsuite.wem application/vnd.mozilla.xul+xml xul application/vnd.ms-artgalry application/vnd.ms-asf application/vnd.ms-excel xls xlb xlt application/vnd.ms-excel.addin.macroEnabled.12 xlam application/vnd.ms-excel.sheet.binary.macroEnabled.12 xlsb application/vnd.ms-excel.sheet.macroEnabled.12 xlsm application/vnd.ms-excel.template.macroEnabled.12 xltm application/vnd.ms-fontobject eot application/vnd.ms-lrm application/vnd.ms-officetheme thmx application/vnd.ms-pki.seccat cat #application/vnd.ms-pki.stl stl application/vnd.ms-powerpoint ppt pps application/vnd.ms-powerpoint.addin.macroEnabled.12 ppam application/vnd.ms-powerpoint.presentation.macroEnabled.12 pptm application/vnd.ms-powerpoint.slide.macroEnabled.12 sldm application/vnd.ms-powerpoint.slideshow.macroEnabled.12 ppsm application/vnd.ms-powerpoint.template.macroEnabled.12 potm application/vnd.ms-project application/vnd.ms-tnef application/vnd.ms-word.document.macroEnabled.12 docm application/vnd.ms-word.template.macroEnabled.12 dotm application/vnd.ms-works application/vnd.mseq application/vnd.msign application/vnd.music-niff application/vnd.musician application/vnd.netfpx application/vnd.noblenet-directory application/vnd.noblenet-sealer application/vnd.noblenet-web application/vnd.novadigm.EDM application/vnd.novadigm.EDX application/vnd.novadigm.EXT application/vnd.oasis.opendocument.chart odc application/vnd.oasis.opendocument.database odb application/vnd.oasis.opendocument.formula odf application/vnd.oasis.opendocument.graphics odg application/vnd.oasis.opendocument.graphics-template otg application/vnd.oasis.opendocument.image odi application/vnd.oasis.opendocument.presentation odp application/vnd.oasis.opendocument.presentation-template otp application/vnd.oasis.opendocument.spreadsheet ods application/vnd.oasis.opendocument.spreadsheet-template ots application/vnd.oasis.opendocument.text odt application/vnd.oasis.opendocument.text-master odm application/vnd.oasis.opendocument.text-template ott application/vnd.oasis.opendocument.text-web oth application/vnd.openxmlformats-officedocument.presentationml.presentation pptx application/vnd.openxmlformats-officedocument.presentationml.slide sldx application/vnd.openxmlformats-officedocument.presentationml.slideshow ppsx application/vnd.openxmlformats-officedocument.presentationml.template potx application/vnd.openxmlformats-officedocument.spreadsheetml.sheet xlsx application/vnd.openxmlformats-officedocument.spreadsheetml.template xltx application/vnd.openxmlformats-officedocument.wordprocessingml.document docx application/vnd.openxmlformats-officedocument.wordprocessingml.template dotx application/vnd.osa.netdeploy application/vnd.palm application/vnd.pg.format application/vnd.pg.osasli application/vnd.powerbuilder6 application/vnd.powerbuilder6-s application/vnd.powerbuilder7 application/vnd.powerbuilder7-s application/vnd.powerbuilder75 application/vnd.powerbuilder75-s application/vnd.previewsystems.box application/vnd.publishare-delta-tree application/vnd.pvi.ptid1 application/vnd.pwg-xhtml-print+xml application/vnd.rapid application/vnd.rim.cod cod application/vnd.s3sms application/vnd.seemail application/vnd.shana.informed.formdata application/vnd.shana.informed.formtemplate application/vnd.shana.informed.interchange application/vnd.shana.informed.package application/vnd.smaf mmf application/vnd.sss-cod application/vnd.sss-dtf application/vnd.sss-ntf application/vnd.stardivision.calc sdc application/vnd.stardivision.chart sds application/vnd.stardivision.draw sda application/vnd.stardivision.impress sdd application/vnd.stardivision.math sdf application/vnd.stardivision.writer sdw application/vnd.stardivision.writer-global sgl application/vnd.street-stream application/vnd.sun.xml.calc sxc application/vnd.sun.xml.calc.template stc application/vnd.sun.xml.draw sxd application/vnd.sun.xml.draw.template std application/vnd.sun.xml.impress sxi application/vnd.sun.xml.impress.template sti application/vnd.sun.xml.math sxm application/vnd.sun.xml.writer sxw application/vnd.sun.xml.writer.global sxg application/vnd.sun.xml.writer.template stw application/vnd.svd application/vnd.swiftview-ics application/vnd.symbian.install sis application/vnd.tcpdump.pcap cap pcap application/vnd.triscape.mxs application/vnd.trueapp application/vnd.truedoc application/vnd.tve-trigger application/vnd.ufdl application/vnd.uplanet.alert application/vnd.uplanet.alert-wbxml application/vnd.uplanet.bearer-choice application/vnd.uplanet.bearer-choice-wbxml application/vnd.uplanet.cacheop application/vnd.uplanet.cacheop-wbxml application/vnd.uplanet.channel application/vnd.uplanet.channel-wbxml application/vnd.uplanet.list application/vnd.uplanet.list-wbxml application/vnd.uplanet.listcmd application/vnd.uplanet.listcmd-wbxml application/vnd.uplanet.signal application/vnd.vcx application/vnd.vectorworks application/vnd.vidsoft.vidconference application/vnd.visio vsd vst vsw vss application/vnd.vividence.scriptfile application/vnd.wap.sic application/vnd.wap.slc application/vnd.wap.wbxml wbxml application/vnd.wap.wmlc wmlc application/vnd.wap.wmlscriptc wmlsc application/vnd.webturbo application/vnd.wordperfect wpd application/vnd.wordperfect5.1 wp5 application/vnd.wrq-hp3000-labelled application/vnd.wt.stf application/vnd.xara application/vnd.xfdl application/vnd.yellowriver-custom-menu application/zlib application/x-123 wk application/x-7z-compressed 7z application/x-abiword abw application/x-apple-diskimage dmg application/x-bcpio bcpio application/x-bittorrent torrent application/x-cab cab application/x-cbr cbr application/x-cbz cbz application/x-cdf cdf cda application/x-cdlink vcd application/x-chess-pgn pgn application/x-comsol mph application/x-core application/x-cpio cpio application/x-csh csh application/x-debian-package deb udeb application/x-director dcr dir dxr application/x-dms dms application/x-doom wad application/x-dvi dvi application/x-executable application/x-font pfa pfb gsf application/x-font-pcf pcf pcf.Z application/x-freemind mm application/x-futuresplash spl application/x-ganttproject gan application/x-gnumeric gnumeric application/x-go-sgf sgf application/x-graphing-calculator gcf application/x-gtar gtar application/x-gtar-compressed tgz taz application/x-hdf hdf #application/x-httpd-eruby rhtml #application/x-httpd-php phtml pht php #application/x-httpd-php-source phps #application/x-httpd-php3 php3 #application/x-httpd-php3-preprocessed php3p #application/x-httpd-php4 php4 #application/x-httpd-php5 php5 application/x-hwp hwp application/x-ica ica application/x-info info application/x-internet-signup ins isp application/x-iphone iii application/x-iso9660-image iso application/x-jam jam application/x-java-applet application/x-java-bean application/x-java-jnlp-file jnlp application/x-jmol jmz application/x-kchart chrt application/x-kdelnk application/x-killustrator kil application/x-koan skp skd skt skm application/x-kpresenter kpr kpt application/x-kspread ksp application/x-kword kwd kwt application/x-latex latex application/x-lha lha application/x-lyx lyx application/x-lzh lzh application/x-lzx lzx application/x-maker frm maker frame fm fb book fbdoc application/x-mif mif application/x-mpegURL m3u8 application/x-ms-application application application/x-ms-manifest manifest application/x-ms-wmd wmd application/x-ms-wmz wmz application/x-msdos-program com exe bat dll application/x-msi msi application/x-netcdf nc application/x-ns-proxy-autoconfig pac application/x-nwc nwc application/x-object o application/x-oz-application oza application/x-pkcs7-certreqresp p7r application/x-pkcs7-crl crl application/x-python-code pyc pyo application/x-qgis qgs shp shx application/x-quicktimeplayer qtl application/x-rdp rdp application/x-redhat-package-manager rpm application/x-rss+xml rss application/x-ruby rb application/x-rx application/x-scilab sci sce application/x-scilab-xcos xcos application/x-sh sh application/x-shar shar application/x-shellscript application/x-shockwave-flash swf swfl application/x-silverlight scr application/x-sql sql application/x-stuffit sit sitx application/x-sv4cpio sv4cpio application/x-sv4crc sv4crc application/x-tar tar application/x-tcl tcl application/x-tex-gf gf application/x-tex-pk pk application/x-texinfo texinfo texi application/x-trash ~ % bak old sik application/x-troff t tr roff application/x-troff-man man application/x-troff-me me application/x-troff-ms ms application/x-ustar ustar application/x-videolan application/x-wais-source src application/x-wingz wz application/x-x509-ca-cert crt application/x-xcf xcf application/x-xfig fig application/x-xpinstall xpi application/x-xz xz audio/32kadpcm audio/3gpp audio/amr amr audio/amr-wb awb audio/annodex axa audio/basic au snd audio/csound csd orc sco audio/flac flac audio/g.722.1 audio/l16 audio/midi mid midi kar audio/mp4a-latm audio/mpa-robust audio/mpeg mpga mpega mp2 mp3 m4a audio/mpegurl m3u audio/ogg oga ogg opus spx audio/parityfec audio/prs.sid sid audio/telephone-event audio/tone audio/vnd.cisco.nse audio/vnd.cns.anp1 audio/vnd.cns.inf1 audio/vnd.digital-winds audio/vnd.everad.plj audio/vnd.lucent.voice audio/vnd.nortel.vbk audio/vnd.nuera.ecelp4800 audio/vnd.nuera.ecelp7470 audio/vnd.nuera.ecelp9600 audio/vnd.octel.sbc audio/vnd.qcelp audio/vnd.rhetorex.32kadpcm audio/vnd.vmx.cvsd audio/x-aiff aif aiff aifc audio/x-gsm gsm audio/x-mpegurl m3u audio/x-ms-wma wma audio/x-ms-wax wax audio/x-pn-realaudio-plugin audio/x-pn-realaudio ra rm ram audio/x-realaudio ra audio/x-scpls pls audio/x-sd2 sd2 audio/x-wav wav chemical/x-alchemy alc chemical/x-cache cac cache chemical/x-cache-csf csf chemical/x-cactvs-binary cbin cascii ctab chemical/x-cdx cdx chemical/x-cerius cer chemical/x-chem3d c3d chemical/x-chemdraw chm chemical/x-cif cif chemical/x-cmdf cmdf chemical/x-cml cml chemical/x-compass cpa chemical/x-crossfire bsd chemical/x-csml csml csm chemical/x-ctx ctx chemical/x-cxf cxf cef #chemical/x-daylight-smiles smi chemical/x-embl-dl-nucleotide emb embl chemical/x-galactic-spc spc chemical/x-gamess-input inp gam gamin chemical/x-gaussian-checkpoint fch fchk chemical/x-gaussian-cube cub chemical/x-gaussian-input gau gjc gjf chemical/x-gaussian-log gal chemical/x-gcg8-sequence gcg chemical/x-genbank gen chemical/x-hin hin chemical/x-isostar istr ist chemical/x-jcamp-dx jdx dx chemical/x-kinemage kin chemical/x-macmolecule mcm chemical/x-macromodel-input mmd mmod chemical/x-mdl-molfile mol chemical/x-mdl-rdfile rd chemical/x-mdl-rxnfile rxn chemical/x-mdl-sdfile sd sdf chemical/x-mdl-tgf tgf #chemical/x-mif mif chemical/x-mmcif mcif chemical/x-mol2 mol2 chemical/x-molconn-Z b chemical/x-mopac-graph gpt chemical/x-mopac-input mop mopcrt mpc zmt chemical/x-mopac-out moo chemical/x-mopac-vib mvb chemical/x-ncbi-asn1 asn chemical/x-ncbi-asn1-ascii prt ent chemical/x-ncbi-asn1-binary val aso chemical/x-ncbi-asn1-spec asn chemical/x-pdb pdb ent chemical/x-rosdal ros chemical/x-swissprot sw chemical/x-vamas-iso14976 vms chemical/x-vmd vmd chemical/x-xtel xtel chemical/x-xyz xyz image/cgm image/g3fax image/gif gif image/ief ief image/jp2 jp2 jpg2 image/jpeg jpeg jpg jpe image/jpm jpm image/jpx jpx jpf image/naplps image/pcx pcx image/png png image/prs.btif image/prs.pti image/svg+xml svg svgz image/tiff tiff tif image/vnd.cns.inf2 image/vnd.djvu djvu djv image/vnd.dwg image/vnd.dxf image/vnd.fastbidsheet image/vnd.fpx image/vnd.fst image/vnd.fujixerox.edmics-mmr image/vnd.fujixerox.edmics-rlc image/vnd.microsoft.icon ico image/vnd.mix image/vnd.net-fpx image/vnd.svf image/vnd.wap.wbmp wbmp image/vnd.xiff image/x-canon-cr2 cr2 image/x-canon-crw crw image/x-cmu-raster ras image/x-coreldraw cdr image/x-coreldrawpattern pat image/x-coreldrawtemplate cdt image/x-corelphotopaint cpt image/x-epson-erf erf image/x-icon image/x-jg art image/x-jng jng image/x-ms-bmp bmp image/x-nikon-nef nef image/x-olympus-orf orf image/x-photoshop psd image/x-portable-anymap pnm image/x-portable-bitmap pbm image/x-portable-graymap pgm image/x-portable-pixmap ppm image/x-rgb rgb image/x-xbitmap xbm image/x-xpixmap xpm image/x-xwindowdump xwd inode/chardevice inode/blockdevice inode/directory-locked inode/directory inode/fifo inode/socket message/delivery-status message/disposition-notification message/external-body message/http message/s-http message/news message/partial message/rfc822 eml model/iges igs iges model/mesh msh mesh silo model/vnd.dwf model/vnd.flatland.3dml model/vnd.gdl model/vnd.gs-gdl model/vnd.gtw model/vnd.mts model/vnd.vtu model/vrml wrl vrml model/x3d+vrml x3dv model/x3d+xml x3d model/x3d+binary x3db multipart/alternative multipart/appledouble multipart/byteranges multipart/digest multipart/encrypted multipart/form-data multipart/header-set multipart/mixed multipart/parallel multipart/related multipart/report multipart/signed multipart/voice-message text/cache-manifest appcache text/calendar ics icz text/css css text/csv csv text/directory text/english text/enriched text/h323 323 text/html html htm shtml text/iuls uls text/mathml mml text/markdown md markdown text/parityfec text/plain asc txt text pot brf srt text/prs.lines.tag text/rfc822-headers text/richtext rtx text/rtf text/scriptlet sct wsc text/t140 text/texmacs tm text/tab-separated-values tsv text/turtle ttl text/uri-list text/vcard vcf vcard text/vnd.abc text/vnd.curl text/vnd.debian.copyright text/vnd.DMClientScript text/vnd.flatland.3dml text/vnd.fly text/vnd.fmi.flexstor text/vnd.in3d.3dml text/vnd.in3d.spot text/vnd.IPTC.NewsML text/vnd.IPTC.NITF text/vnd.latex-z text/vnd.motorola.reflex text/vnd.ms-mediapackage text/vnd.sun.j2me.app-descriptor jad text/vnd.wap.si text/vnd.wap.sl text/vnd.wap.wml wml text/vnd.wap.wmlscript wmls text/x-bibtex bib text/x-boo boo text/x-c++hdr h++ hpp hxx hh text/x-c++src c++ cpp cxx cc text/x-chdr h text/x-component htc text/x-crontab text/x-csh csh text/x-csrc c text/x-dsrc d text/x-diff diff patch text/x-haskell hs text/x-java java text/x-lilypond ly text/x-literate-haskell lhs text/x-makefile text/x-moc moc text/x-pascal p pas text/x-pcs-gcd gcd text/x-perl pl pm text/x-python py text/x-scala scala text/x-server-parsed-html text/x-setext etx text/x-sfv sfv text/x-sh sh text/x-tcl tcl tk text/x-tex tex ltx sty cls text/x-vcalendar vcs video/3gpp 3gp video/annodex axv video/dl dl video/dv dif dv video/fli fli video/gl gl video/mpeg mpeg mpg mpe video/MP2T ts video/mp4 mp4 video/quicktime qt mov video/mp4v-es video/ogg ogv video/parityfec video/pointer video/webm webm video/vnd.fvt video/vnd.motorola.video video/vnd.motorola.videop video/vnd.mpegurl mxu video/vnd.mts video/vnd.nokia.interleaved-multimedia video/vnd.vivo video/x-flv flv video/x-la-asf lsf lsx video/x-mng mng video/x-ms-asf asf asx video/x-ms-wm wm video/x-ms-wmv wmv video/x-ms-wmx wmx video/x-ms-wvx wvx video/x-msvideo avi video/x-sgi-movie movie video/x-matroska mpv mkv x-conference/x-cooltalk ice x-epoc/x-sisx-app sisx x-world/x-vrml vrm vrml wrl Events: <none> [root@vm0 ceph]#
3. StorageClassの作成
CephObjectStoreを作成したので、次にバケットを作成します。バケットを作成するには、StorageClassリソースを作成し、それを後述のObjectBucketClaimが利用することで作成されます。
parameters:StorageClassのパラメータを指定
※参考リンク：
Kubernetes Doc - Storage Classes
サンプル例
以下のyamlファイルでは、作成済みのCephObjectStoreの情報を指定します。
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-ceph-delete-bucket provisioner: ceph.rook.io/bucket reclaimPolicy: Delete parameters: objectStoreName: my-store objectStoreNamespace: rook-ceph region: us-east-1
上記ファイルを利用してStorageClassを作成します。
[root@vm0 ceph]# kubectl apply -f storageclass-bucket-delete.yaml storageclass.storage.k8s.io/rook-ceph-delete-bucket created [root@vm0 ceph]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE rook-ceph-delete-bucket ceph.rook.io/bucket Delete Immediate false 41s
4. ObjectBucketClaimの作成
StorageClass作成後、バケットを作成するためにObjectBucketClaimを作成します。KubernetesにおけるStorage ClassとPersistent Volume Claimとの関係に似ています。
spec.bucketName: バケット名を指定。新規で作成する場合、オブジェクトストア全体で一意のバケット名を与える必要があるため、推奨されないspec.generateBucketName: ランダムに生成されるバケット名のprefix値を指定。spec.bucketNameが指定される場合、spec.generateBucketNameの値は無視される。どちらも空欄の場合はStorageClassでバケット名が指定されている場合であるspec.storageClassName: 利用するStorageClassの名前を指定spec.additionalConfig: 特定のbucket provisionerを利用する場合などに利用。バケットのprovisioning時に利用する値をkey-value形式で指定
サンプル例
以下のyamlファイルでは、generateBucketNameを利用してバケット名を生成します。
apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: ceph-delete-bucket spec: generateBucketName: ceph-bkt storageClassName: rook-ceph-delete-bucket
上記ファイルを利用してリソースを作成します。
[root@vm0 ceph]# kubectl apply -f object-bucket-claim-delete.yaml objectbucketclaim.objectbucket.io/ceph-delete-bucket created [root@vm0 ceph]# kubectl get objectbucketclaim.objectbucket.io NAME AGE ceph-delete-bucket 15s
ObjectBucketClaimリソースを用意しバケットが作られると、バケットを利用するための情報を格納したSecretとConfigMapが自動的に作成されます。
# SecretとConfigMapが作成される [root@vm0 ceph]# kubectl get secret NAME TYPE DATA AGE ceph-delete-bucket Opaque 2 62s default-token-d4jbr kubernetes.io/service-account-token 3 16d [root@vm0 ceph]# kubectl get cm NAME DATA AGE ceph-delete-bucket 6 92s
※参考リンク：
Kubernetes Doc - Configure a Pod to Use a ConfigMap
5. オブジェクトストレージの利用
アプリケーションから作成したバケットを利用するため、ConfigMapとSecretから必要な情報を取得します。ここでは、toolbox Podからs3cmdを実行してバケットを利用するため、以下の情報を取得します。
# ConfigMapの情報 [root@vm0 ceph]# kubectl get cm ceph-delete-bucket -o yaml apiVersion: v1 data: BUCKET_HOST: rook-ceph-rgw-my-store.rook-ceph BUCKET_NAME: ceph-bkt-4db5fb92-a847-4958-b899-f88af8b5f08e BUCKET_PORT: "80" BUCKET_REGION: us-east-1 BUCKET_SSL: "false" BUCKET_SUBREGION: "" kind: ConfigMap metadata: creationTimestamp: "2020-01-31T04:31:49Z" finalizers: - objectbucket.io/finalizer name: ceph-delete-bucket namespace: default ownerReferences: - apiVersion: objectbucket.io/v1alpha1 blockOwnerDeletion: true controller: true kind: ObjectBucketClaim name: ceph-delete-bucket uid: 7a4f00c0-6110-4c5c-bf35-a5869598a00f resourceVersion: "1004856" selfLink: /api/v1/namespaces/default/configmaps/ceph-delete-bucket uid: 19455f06-1d1e-400e-bb01-f9d92f64adb1 [root@vm0 ceph]# #Secretの情報 [root@vm0 ceph]# kubectl get secret ceph-delete-bucket -o yaml apiVersion: v1 data: AWS_ACCESS_KEY_ID: WjZKUEpOQTE2OTdKRVJFNVo2M0o= AWS_SECRET_ACCESS_KEY: czFUVHZRa0lyYlVNRWZYczRUWElOemd1cVQ3MFFCSGo1NEtMZTNISg== kind: Secret metadata: creationTimestamp: "2020-01-31T04:31:49Z" finalizers: - objectbucket.io/finalizer name: ceph-delete-bucket namespace: default ownerReferences: - apiVersion: objectbucket.io/v1alpha1 blockOwnerDeletion: true controller: true kind: ObjectBucketClaim name: ceph-delete-bucket uid: 7a4f00c0-6110-4c5c-bf35-a5869598a00f resourceVersion: "1004855" selfLink: /api/v1/namespaces/default/secrets/ceph-delete-bucket uid: 8a5ef469-89cc-4f9a-b30d-0e7eff2b4a11 type: Opaque [root@vm0 ceph]# # AWS_ACCESS_KEY_ID [root@vm0 ceph]# echo WjZKUEpOQTE2OTdKRVJFNVo2M0o= | base64 --decode Z6JPJNA1697JERE5Z63J # AWS_SECRET_ACCESS_KEY [root@vm0 ceph]# echo czFUVHZRa0lyYlVNRWZYczRUWElOemd1cVQ3MFFCSGo1NEtMZTNISg== | base64 --decode s1TTvQkIrbUMEfXs4TXINzguqT70QBHj54KLe3HJ # AWS_ENDPOINT [root@vm0 ceph]# kubectl get svc -n rook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE csi-cephfsplugin-metrics ClusterIP 10.96.42.224 <none> 8080/TCP,8081/TCP 23m csi-rbdplugin-metrics ClusterIP 10.96.154.224 <none> 8080/TCP,8081/TCP 23m rook-ceph-mgr ClusterIP 10.96.152.170 <none> 9283/TCP 22m rook-ceph-mgr-dashboard ClusterIP 10.96.50.233 <none> 8443/TCP 22m rook-ceph-mon-a ClusterIP 10.96.219.41 <none> 6789/TCP,3300/TCP 22m rook-ceph-rgw-my-store ClusterIP 10.96.123.60 <none> 80/TCP 19m
上記情報を利用し、Toolbox Podからバケットにアクセスします。バケットにアクセスするため、s3cmdで必要な設定を行います。
# s3cmdインストール [root@vm0 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-7f96779fb9-kjvpc -- yum --assumeyes install s3cmd [root@vm0 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-7f96779fb9-kjvpc -- s3cmd --version s3cmd version 2.0.2 # テスト用ファイルの作成 [root@vm0 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-7f96779fb9-kjvpc /bin/sh sh-4.2# sh-4.2# echo "Hello Rook" > /tmp/rookObj sh-4.2# ls /tmp/ rookObj # 環境変数の設定 sh-4.2# export AWS_HOST=rook-ceph-rgw-my-store.rook-ceph sh-4.2# export AWS_ENDPOINT=10.96.123.60:80 sh-4.2# export AWS_ACCESS_KEY_ID=Z6JPJNA1697JERE5Z63J sh-4.2# export AWS_SECRET_ACCESS_KEY=s1TTvQkIrbUMEfXs4TXINzguqT70QBHj54KLe3HJ # コンフィグの設定（必要に応じて） sh-4.2# s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key [Z6JPJNA1697JERE5Z63J]: Secret Key [s1TTvQkIrbUMEfXs4TXINzguqT70QBHj54KLe3HJ]: Default Region [US]: Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: 10.96.123.60:80 Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: rook-ceph-rgw-my-store.rook-ceph Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]: no On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name: New settings: Access Key: Z6JPJNA1697JERE5Z63J Secret Key: s1TTvQkIrbUMEfXs4TXINzguqT70QBHj54KLe3HJ Default Region: US S3 Endpoint: 10.96.123.60:80 DNS-style bucket+hostname:port template for accessing a bucket: rook-ceph-rgw-my-store.rook-ceph Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Y Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Not configured. Never mind. Save settings? [y/N] y Configuration saved to '/root/.s3cfg' sh-4.2#
ここからs3cmdを用いてバケットにアクセスし、テスト用ファイルの配置と取得を行います。
# バケットの確認 sh-4.2# s3cmd ls 2020-01-31 04:31 s3://ceph-bkt-4db5fb92-a847-4958-b899-f88af8b5f08e # テスト用ファイルの配置 sh-4.2# s3cmd put /tmp/rookObj --no-ssl --host=${AWS_HOST} --host-bucket= s3://ceph-bkt-4db5fb92-a847-4958-b899-f88af8b5f08e upload: '/tmp/rookObj' -> 's3://ceph-bkt-4db5fb92-a847-4958-b899-f88af8b5f08e/rookObj' [1 of 1] 11 of 11 100% in 0s 190.37 B/s done # バケットの確認 sh-4.2# s3cmd ls s3://ceph-bkt-4db5fb92-a847-4958-b899-f88af8b5f08e --no-ssl --host=rook-ceph-rgw-my-store.rook-ceph 2020-01-31 05:00 11 s3://ceph-bkt-4db5fb92-a847-4958-b899-f88af8b5f08e/rookObj # バケット内オブジェクトの取得 sh-4.2# s3cmd get s3://ceph-bkt-4db5fb92-a847-4958-b899-f88af8b5f08e/rookObj /tmp/rookObj-download --no-ssl --host=${AWS_HOST} download: 's3://ceph-bkt-4db5fb92-a847-4958-b899-f88af8b5f08e/rookObj' -> '/tmp/rookObj-download' [1 of 1] 11 of 11 100% in 0s 259.07 B/s done # オブジェクト取得結果の確認 sh-4.2# cat /tmp/rookObj-download Hello Rook
また、以下のようにPod内で用いる環境変数を指定することでバケットを利用することもできます。
apiVersion: v1 kind: Pod metadata: name: testpod spec: containers: - name: redis-container image: redis envFrom: - configMapRef: name: ceph-delete-bucket - secretRef: name: ceph-delete-bucket
# ConfigMap [root@vm0 ceph]# kubectl get cm NAME DATA AGE ceph-delete-bucket 6 70m #Secret [root@vm0 ceph]# kubectl get secret NAME TYPE DATA AGE ceph-delete-bucket Opaque 2 72m default-token-d4jbr kubernetes.io/service-account-token 3 17d # Podの作成 [root@vm0 ceph]# kubectl apply -f object-testpod.yaml pod/testpod created [root@vm0 ceph]# kubectl get pods NAME READY STATUS RESTARTS AGE testpod 1/1 Running 0 14s # Pod内の環境変数確認 [root@vm0 ceph]# kubectl exec -it testpod -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=testpod TERM=xterm BUCKET_REGION=us-east-1 BUCKET_SSL=false BUCKET_SUBREGION= BUCKET_HOST=rook-ceph-rgw-my-store.rook-ceph AWS_ACCESS_KEY_ID=Z6JPJNA1697JERE5Z63J AWS_SECRET_ACCESS_KEY=s1TTvQkIrbUMEfXs4TXINzguqT70QBHj54KLe3HJ BUCKET_NAME=ceph-bkt-4db5fb92-a847-4958-b899-f88af8b5f08e BUCKET_PORT=80 KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT=tcp://10.96.0.1:443 KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 KUBERNETES_SERVICE_HOST=10.96.0.1 KUBERNETES_SERVICE_PORT=443 GOSU_VERSION=1.11 REDIS_VERSION=5.0.7 REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-5.0.7.tar.gz REDIS_DOWNLOAD_SHA=61db74eabf6801f057fd24b590232f2f337d422280fd19486eca03be87d3a82b HOME=/root [root@vm0 ceph]#
その他
Erasure Code
CephObjectStoreのデータ冗長方式をerasureCodedにした場合も載せておきます。
apiVersion: ceph.rook.io/v1 kind: CephObjectStore metadata: name: my-store namespace: rook-ceph spec: # The pool spec used to create the metadata pools. Must use replication. metadataPool: failureDomain: host replicated: size: 3 # The pool spec used to create the data pool. Can use replication or erasure coding. dataPool: failureDomain: host erasureCoded: dataChunks: 2 codingChunks: 1 # Whether to preserve metadata and data pools on object store deletion preservePoolsOnDelete: true # The gateway service configuration gateway: # type of the gateway (s3) type: s3 # A reference to the secret in the rook namespace where the ssl certificate is stored sslCertificateRef: # The port that RGW pods will listen on (http) port: 80 # The port that RGW pods will listen on (https). An ssl certificate is required. securePort: # The number of pods in the rgw deployment instances: 1
上記ファイルを利用してCephObjectStoreリソースを作成します。
[root@vm0 ceph]# kubectl apply -f object-ec.yaml cephobjectstore.ceph.rook.io/my-store created [root@vm0 ceph]# kubectl get cephobjectstore.ceph.rook.io -n rook-ceph NAME AGE my-store 17s
toolbox Podからceph osd erasure-code-profile ls ceph osd erasure-code-profile getコマンドを実行することで、作成されたErasure Code Profileの内容を確認することができます。
[root@vm0 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-7f96779fb9-kjvpc -- ceph osd erasure-code-profile ls default my-store_ecprofile [root@vm0 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-7f96779fb9-kjvpc -- ceph osd erasure-code-profile get my-store_ecprofile crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8 [root@vm0 ceph]#
※参考リンク：
Ceph Doc - JERASURE ERASURE CODE PLUGIN
Access External to the Cluster
クラスター外からバケットにアクセスする場合は、追加でServiceリソースを作成します。以下のyamlファイルではNodePortタイプのServiceリソースを利用しています。
apiVersion: v1 kind: Service metadata: name: rook-ceph-rgw-my-store-external namespace: rook-ceph labels: app: rook-ceph-rgw rook_cluster: rook-ceph rook_object_store: my-store spec: ports: - name: rgw port: 80 protocol: TCP targetPort: 80 selector: app: rook-ceph-rgw rook_cluster: rook-ceph rook_object_store: my-store sessionAffinity: None type: NodePort
[root@vm0 ceph]# kubectl apply -f rgw-external.yaml service/rook-ceph-rgw-my-store-external created [root@vm0 ceph]# [root@vm0 ceph]# kubectl get svc -n rook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE csi-cephfsplugin-metrics ClusterIP 10.96.42.224 <none> 8080/TCP,8081/TCP 62m csi-rbdplugin-metrics ClusterIP 10.96.154.224 <none> 8080/TCP,8081/TCP 62m rook-ceph-mgr ClusterIP 10.96.152.170 <none> 9283/TCP 61m rook-ceph-mgr-dashboard ClusterIP 10.96.50.233 <none> 8443/TCP 61m rook-ceph-mon-a ClusterIP 10.96.219.41 <none> 6789/TCP,3300/TCP 62m rook-ceph-rgw-my-store ClusterIP 10.96.123.60 <none> 80/TCP 58m rook-ceph-rgw-my-store-external NodePort 10.96.224.160 <none> 80:30270/TCP 10s ★
※参考リンク：
Kubernetes Doc - Service # Publishing Services (ServiceTypes)
Create a User
オブジェクトストア内でで利用するユーザーをCephObjectStoreUser CRDから作成することができます。
spec.store: ユーザーが作成されるオブジェクトストア名を指定spec.displayName: 表示名を指定
以下のyamlファイルを利用して作成します。
apiVersion: ceph.rook.io/v1 kind: CephObjectStoreUser metadata: name: my-user namespace: rook-ceph spec: store: my-store displayName: "my display name"
上記ファイルを利用してリソースを作成します。
[root@vm0 ceph]# kubectl apply -f object-user.yaml cephobjectstoreuser.ceph.rook.io/my-user created [root@vm0 ceph]# kubectl get cephobjectstoreuser.ceph.rook.io -n rook-ceph NAME AGE my-user 16s
上記のようにリソースが作成されると、合わせてSecretリソースも作成されます。SecretにはAccessKey SecretKeyが格納されています。
[root@vm0 ceph]# kubectl get secret -n rook-ceph NAME TYPE DATA AGE default-token-lf9qc kubernetes.io/service-account-token 3 71m rook-ceph-admin-keyring kubernetes.io/rook 1 65m rook-ceph-cmd-reporter-token-4fr68 kubernetes.io/service-account-token 3 71m rook-ceph-config kubernetes.io/rook 2 65m rook-ceph-crash-collector-keyring kubernetes.io/rook 1 65m rook-ceph-dashboard-password kubernetes.io/rook 1 65m rook-ceph-mgr-a-keyring kubernetes.io/rook 1 65m rook-ceph-mgr-token-6zlhm kubernetes.io/service-account-token 3 71m rook-ceph-mon kubernetes.io/rook 4 65m rook-ceph-mons-keyring kubernetes.io/rook 1 65m rook-ceph-object-user-my-store-my-user kubernetes.io/rook 2 25s rook-ceph-osd-0-keyring kubernetes.io/rook 1 64m rook-ceph-osd-1-keyring kubernetes.io/rook 1 64m rook-ceph-osd-2-keyring kubernetes.io/rook 1 64m rook-ceph-osd-token-7kwbg kubernetes.io/service-account-token 3 71m rook-ceph-rgw-my-store-a-keyring kubernetes.io/rook 1 61m rook-ceph-system-token-2d7b7 kubernetes.io/service-account-token 3 71m rook-csi-cephfs-node kubernetes.io/rook 2 65m rook-csi-cephfs-plugin-sa-token-4cjgp kubernetes.io/service-account-token 3 71m rook-csi-cephfs-provisioner kubernetes.io/rook 2 65m rook-csi-cephfs-provisioner-sa-token-94wgm kubernetes.io/service-account-token 3 71m rook-csi-rbd-node kubernetes.io/rook 2 65m rook-csi-rbd-plugin-sa-token-v2zfp kubernetes.io/service-account-token 3 71m rook-csi-rbd-provisioner kubernetes.io/rook 2 65m rook-csi-rbd-provisioner-sa-token-nlhdk kubernetes.io/service-account-token 3 71m [root@vm0 ceph]# kubectl get secret -n rook-ceph rook-ceph-object-user-my-store-my-user -o yaml apiVersion: v1 data: AccessKey: WjJRWllBRzRLQlhRUUVWOTJPNk0= SecretKey: d282OGs0T25oVnBRU0NQOE5YVm01VUR5NjdicllXQ0czT1ROa3BlOA== kind: Secret metadata: creationTimestamp: "2020-01-31T05:22:44Z" labels: app: rook-ceph-rgw rook_cluster: rook-ceph rook_object_store: my-store user: my-user name: rook-ceph-object-user-my-store-my-user namespace: rook-ceph ownerReferences: - apiVersion: ceph.rook.io/v1 blockOwnerDeletion: true kind: CephCluster name: rook-ceph uid: 90b169b7-1969-4fb7-9a52-2076450dfa13 resourceVersion: "1019001" selfLink: /api/v1/namespaces/rook-ceph/secrets/rook-ceph-object-user-my-store-my-user uid: 5fcd2bea-6636-4ae9-a8cd-c43704f020df type: kubernetes.io/rook
toolbox Podからradosgw-admin user infoコマンドを実行することで、作成したユーザー情報を確認することもできます。
# ユーザー情報の確認 [root@vm0 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-7f96779fb9-kjvpc -- radosgw-admin user info --uid=my-user { "user_id": "my-user", "display_name": "my display name", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "my-user", "access_key": "Z2QZYAG4KBXQQEV92O6M", "secret_key": "wo68k4OnhVpQSCP8NXVm5UDy67brYWCG3OTNkpe8" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } [root@vm0 ceph]#
※参考リンク：
Ceph Doc - RADOSGW Admin Guide
疑問点
s3cmdでバケットを作成しようとすると以下のメッセージが。ObjectBucketClaimでの作成のみが可能？
sh-4.2# s3cmd mb --no-ssl --host=${AWS_HOST} --host-bucket= s3://atestbucket ERROR: S3 error: 400 (TooManyBuckets) sh-4.2#
参考ドキュメント
Rook Doc - Ceph Object Store CRD
Rook Doc - Ceph Object Bucket Claim
Rook Doc - Ceph Object Store User CRD
赤帽エンジニアブログ - ストレージオーケストレーター Rook : 第7話 宿命のObject Bucket(Aパート)
赤帽エンジニアブログ - ストレージオーケストレーター Rook : 第8話 宿命のObject Bucket(Bパート)
