Karan Singh

Where there's a Cloud , there's a way !!

Ceph Object Storage : Part-II (Indexless Buckets)

| Comments

Indexless Bucket

This is episode-2 of Ceph Object Storage Blog Series , you can refer to episode-1 here where i have explained the internals of Ceph Object Storage and covering over Ceph Indexless Buckets feature. In this episode we will go through the implementation of Ceph indexless buckets

By default Ceph RGW creates standard indexed bucket ( i.e. non-indexless buckets ). These buckets can list the objects stored in them. Lets verify standard bucket before configuring indexless buckets.

  • In the following snippet we have created a bucket called ìndexed-bucket , added one object into it and then tried to list that bucket which went fine.
1
2
3
4
5
6
7
8
9
[[email protected] ~]# s3cmd mb s3://indexed-bucket
Bucket 's3://indexed-bucket/' created
[[email protected] ~]# s3cmd put /etc/hosts s3://indexed-bucket/hosts
upload: '/etc/hosts' -> 's3://indexed-bucket/hosts'  [1 of 1]
 626 of 626   100% in    2s   255.25 B/s  done
[[email protected] ~]#
[[email protected] ~]# s3cmd ls s3://indexed-bucket
2017-01-15 11:37       626   s3://indexed-bucket/hosts
[[email protected] ~]#
  • As expected indexed-bucket (above) can list objects stored in them.

  • Before configuring indexless buckets, let’s verify realm, zonegroup and zone configuration . One important note here is to make sure the default_info section is populated with ID , if its not set then RGW service will not restart after changing zone configuration file which is required for indexless configuration.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[[email protected] ~]# radosgw-admin realm list
{
    "default_info": "",
    "realms": []
}

[[email protected] ~]#
[[email protected] ~]# radosgw-admin zonegroup list
read_default_id : -2
{
    "default_info": "",
    "zonegroups": [
        "default"
    ]
}

[[email protected] ~]# radosgw-admin zone list
{
    "default_info": "",
    "zones": [
        "default"
    ]
}
  • Since default_info is not populated under realm, let’s generate one.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[[email protected] ~]# radosgw-admin realm create --rgw-realm=default --default
.rgw.root:periods.d69cf4e0-5a70-454e-8096-e582cf7c3a53.latest_epoch
{
    "id": "7d7a5331-fb62-4e1f-97d2-7003767ad7bd",
    "name": "default",
    "current_period": "d69cf4e0-5a70-454e-8096-e582cf7c3a53",
    "epoch": 1
}

[[email protected] tmp]# radosgw-admin realm list
{
    "default_info": "7d7a5331-fb62-4e1f-97d2-7003767ad7bd",
    "realms": [
        "default"
    ]
}
  • Make a note of realm ID , we will add this ID in zonegroup and zone configuration files.
  • Extract current zonegroup and zone configuration file
1
2
3
4
[[email protected] tmp]# radosgw-admin zonegroup get --rgw-zonegroup=default > default-zonegroup.json
[[email protected] tmp]#
[[email protected] tmp]# radosgw-admin zone get --rgw-zone=default > default-zone.json
[[email protected] tmp]#
  • Edit default-zonegroup.json file and add realm ID in realm_id section
1
[[email protected] tmp]# vim default-zonegroup.json
  • Edit default-zone.json file add realm ID in realm_id section and change “index_type”: 0 to “index_type”: 1
  • This index_type: 1 represents indexless bucket , while index_type: 0 represents standard index bucket
1
[[email protected] tmp]# vim default-zone.json
  • Inject back the zonegroup and zone configuration file that we have edited before
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
[[email protected] tmp]# radosgw-admin zonegroup set --rgw-zonegroup=default --infile default-zonegroup.json --default
{
    "id": "7a30db23-a015-491b-b70b-878952a5d535",
    "name": "default",
    "api_name": "",
    "is_master": "true",
    "endpoints": [],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "9f264087-87d0-4be6-9542-09dfec1f87db",
    "zones": [
        {
            "id": "9f264087-87d0-4be6-9542-09dfec1f87db",
            "name": "default",
            "endpoints": [],
            "log_meta": "false",
            "log_data": "false",
            "bucket_index_max_shards": 0,
            "read_only": "false"
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "7d7a5331-fb62-4e1f-97d2-7003767ad7bd"
}
[[email protected] tmp]#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[[email protected] tmp]# radosgw-admin zone set --rgw-zone=default --infile default-zone.json --default
zone id 9f264087-87d0-4be6-9542-09dfec1f87db{
    "id": "9f264087-87d0-4be6-9542-09dfec1f87db",
    "name": "default",
    "domain_root": "default.rgw.data.root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.gc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.intent-log",
    "usage_log_pool": "default.rgw.usage",
    "user_keys_pool": "default.rgw.users.keys",
    "user_email_pool": "default.rgw.users.email",
    "user_swift_pool": "default.rgw.users.swift",
    "user_uid_pool": "default.rgw.users.uid",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "data_pool": "default.rgw.buckets.data",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 1
            }
        }
    ],
    "metadata_heap": "default.rgw.meta",
    "realm_id": "7d7a5331-fb62-4e1f-97d2-7003767ad7bd"
}
[[email protected] tmp]#
  • Now once you list zonegroup and zone configuration you will see default_info field populated
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[[email protected] tmp]# radosgw-admin zonegroup list
read_default_id : 0
{
    "default_info": "7a30db23-a015-491b-b70b-878952a5d535",
    "zonegroups": [
        "default"
    ]
}

[[email protected] tmp]#
[[email protected] tmp]# radosgw-admin zone list
{
    "default_info": "9f264087-87d0-4be6-9542-09dfec1f87db",
    "zones": [
        "default"
    ]
}

[[email protected] tmp]#
  • Now since we have done changes to zone and realm ID , we need to update RGW periods by commiting it
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
[[email protected] tmp]# radosgw-admin period list
{
    "periods": [
        "d69cf4e0-5a70-454e-8096-e582cf7c3a53"
    ]
}

[[email protected] tmp]# radosgw-admin period update --commit --rgw-zone=default
.rgw.root:periods.3113a93d-eefc-40c2-9931-68bd5854ec37.latest_epoch
{
    "id": "3113a93d-eefc-40c2-9931-68bd5854ec37",
    "epoch": 1,
    "predecessor_uuid": "d69cf4e0-5a70-454e-8096-e582cf7c3a53",
    "sync_status": [
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        ""
    ],
    "period_map": {
        "id": "3113a93d-eefc-40c2-9931-68bd5854ec37",
        "zonegroups": [
            {
                "id": "7a30db23-a015-491b-b70b-878952a5d535",
                "name": "default",
                "api_name": "",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "9f264087-87d0-4be6-9542-09dfec1f87db",
                "zones": [
                    {
                        "id": "9f264087-87d0-4be6-9542-09dfec1f87db",
                        "name": "default",
                        "endpoints": [],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "7d7a5331-fb62-4e1f-97d2-7003767ad7bd"
            }
        ],
        "short_zone_ids": [
            {
                "key": "9f264087-87d0-4be6-9542-09dfec1f87db",
                "val": 2479689693
            }
        ]
    },
    "master_zonegroup": "7a30db23-a015-491b-b70b-878952a5d535",
    "master_zone": "9f264087-87d0-4be6-9542-09dfec1f87db",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "max_size_kb": -1,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "max_size_kb": -1,
            "max_objects": -1
        }
    },
    "realm_id": "7d7a5331-fb62-4e1f-97d2-7003767ad7bd",
    "realm_name": "default",
    "realm_epoch": 2
}
[[email protected] tmp]#

[[email protected] tmp]# radosgw-admin period list
{
    "periods": [
        "3113a93d-eefc-40c2-9931-68bd5854ec37",
        "7d7a5331-fb62-4e1f-97d2-7003767ad7bd:staging",
        "d69cf4e0-5a70-454e-8096-e582cf7c3a53"
    ]
}
[[email protected] tmp]#
  • Restart ceph-radosgw services
1
2
3
4
5
6
7
8
9
10
11
12
13
[[email protected] tmp]# systemctl restart [email protected]
[[email protected] tmp]#
[[email protected] tmp]# systemctl status [email protected]
[email protected] - Ceph rados gateway
   Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2017-01-15 13:52:17 EET; 13s ago
 Main PID: 642609 (radosgw)
   CGroup: /system.slice/system-ceph\[email protected]
           └─642609 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-rgw1 --setuser ceph --setgroup ceph

Jan 15 13:52:17 ceph-rgw1 systemd[1]: Started Ceph rados gateway.
Jan 15 13:52:17 ceph-rgw1 systemd[1]: Starting Ceph rados gateway...
[[email protected] tmp]#
  • Finally let’s verify indexless buckets , create a bucket called as indexless-bucket
1
2
3
[[email protected] tmp]# s3cmd mb s3://indexless-bucket
Bucket 's3://indexless-bucket/' created
[[email protected] tmp]#
  • Put object in the bucket
1
2
3
[[email protected] tmp]#  s3cmd put /etc/hosts s3://indexless-bucket/hosts
upload: '/etc/hosts' -> 's3://indexless-bucket/hosts'  [1 of 1]
 626 of 626   100% in    0s   101.28 kB/s  done
  • List bucket and you will find that this bucket will not list objects as its configured as indexless bucket and does not have index/metadata information to list objects.
1
2
[[email protected] tmp]# s3cmd ls s3://indexless-bucket
[[email protected] tmp]#

Here you go !!! Now you have RGW configured for indexless buckets.

Its interesting to note that you can configure both standard bucket / indexless bucket in the same cluster but to a different zone and zonegroup. It should be trivial to create a new zone and zonegroup with support to indexless bucket while the default zone/zonegroup should still point to standard indexed buckets. ( I think this is a good experiment to try , once i get time )

Comments