opentsdb探索之路——部分设计与实现 (5)

QueryUtil#getRowKeyUIDRegex()

// Generate a regexp for our tags. Say we have 2 tags: { 0 0 1 0 0 2 } // and { 4 5 6 9 8 7 }, the regexp will be: // "^.{7}(?:.{6})*\\Q\000\000\001\000\000\002\\E(?:.{6})*\\Q\004\005\006\011\010\007\\E(?:.{6})*$"

官方对查询设置explicit_tags为true的介绍:
#explicit-tags
意思我已经知道了要查询的metric明确只有这些tags,想查询的时序数据不会出现其他tag,这样opentsdb就会把用户过滤的tag直接拼到rowkey中,一定程度上优化了查询。见代码

if (explicit_tags && enable_fuzzy_filter) { fuzzy_key = new byte[prefix_width + (row_key_literals.size() * (name_width + value_width))]; fuzzy_mask = new byte[prefix_width + (row_key_literals.size() * (name_width + value_width))]; System.arraycopy(scanner.getCurrentKey(), 0, fuzzy_key, 0, scanner.getCurrentKey().length); // 因为已经明确了只有哪些指定的tag,这个时候才会把tags直接拼到startKey中 scanner.setStartKey(fuzzy_key); }

explicit_tags为true的情况下,会用FuzzyRowFilter,看一下源码中的描述

/** * FuzzyRowFilter is a server-side fast-forward filter that allows skipping * whole range of rows when scanning. The feature is available in HBase * 0.94.5 and above. * <p> * It takes two byte array to match a rowkey, one to hold the fixed value * and one to hold a mask indicating which bytes of the rowkey must match the * fixed value. The two arrays must have the same length. * <p> * Bytes in the mask can take two values, 0 meaning that the corresponding byte * in the rowkey must match the corresponding fixed byte and 1 meaning that the * corresponding byte in the rowkey can take any value. * <p> * One can combine several {@link FuzzyFilterPair} to match multiple patterns at * once. * <p> * Example : * You store logs with this rowkey design : * group(3bytes)timestamp(4bytes)severity(1byte) * * You want to get all FATAL("5") logs : * * Build a FuzzyFilterPair with * - rowkey : "????????5" * - fuzzy mask : "111111110" * And CRITICAL("4") logs only for web servers : * * Add another FuzzyFilterPair with * - rowkey : "web????4" * - fuzzy mask : "00011110" * * @since 1.7 */ public final class FuzzyRowFilter extends ScanFilter { // ... }

总结一下就是,如果你明确你要查的数据有哪几个tag,建议查询的时候指定explicit_tags为true,有助于查询优化。

# Example 1: :4242/api/query?start=1h-ago&m=sum:explicit_tags:sys.cpu.system{host=web01} # Example 2: :4242/api/query?start=1h-ago&m=sum:explicit_tags:sys.cpu.system{host=*}{dc=*} # Example 3: { "start":1584408560754, "end":1584409460754, "msResolution":false, "queries":[ { "aggregator":"avg", "metric":"metric.test", "downsample":"5m-avg", "explicitTags":true, "filters":[ { "type":"literal_or", "tagk":"instance", "filter":"total", "groupBy":true }, { "type":"literal_or", "tagk":"ip", "filter":"192.168.1.1", "groupBy":true } ] } ] }

关于tsd.storage.use_otsdb_timestamp这个配置与HBase特性有关。下篇写优化的时候再讲,这里提出来放在这里。TsdbQuery#getScanner(final int salt_bucket)

// tsd.storage.use_otsdb_timestamp if (tsdb.getConfig().use_otsdb_timestamp()) { long stTime = (getScanStartTimeSeconds() * 1000); long endTime = end_time == UNSET ? -1 : (getScanEndTimeSeconds() * 1000); if (tsdb.getConfig().get_date_tiered_compaction_start() <= stTime && rollup_query == null) { // TODO - we could set this for rollups but we also need to write // the rollup columns at the proper time. scanner.setTimeRange(stTime, endTime); } } rowkey中加salt的情况(Salting)

相关文档:
#salting

时序数据的写入,写热点是一个不可规避的问题,当某个metric下数据点很多时,则该metric很容易造成写入热点,即往一个region server写,甚至同一个region,如果这样,对这部分数据的读写都会落到HBase集群中的一台机器上,无法发挥集群的处理能力,甚至直接将某个region server压垮。加salt就是为了将时序数据的rowkey打散,从而分配到不同的region中,以均衡负载。

When enabled, a configured number of bytes are prepended to each row key. Each metric and combination of tags is then hashed into one "bucket", the ID of which is written to the salt bytes
从2.2开始,OpenTSDB采取了允许将metric加salt,加salt后的变化就是在rowkey前会拼上一个桶编号(bucket index)。

To enable salting you must modify the config file parameter tsd.storage.salt.width and optionally tsd.storage.salt.buckets. We recommend setting the salt width to 1 and determine the number of buckets based on a factor of the number of region servers in your cluster. Note that at query time, the TSD will fire tsd.storage.salt.buckets number of scanners to fetch data. The proper number of salt buckets must be determined through experimentation as at some point query performance may suffer due to having too many scanners open and collating the results. In the future the salt width and buckets may be configurable but we didn't want folks changing settings on accident and losing data.

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/wsfffd.html